2019-12-11T16:13:36.860 INFO:root:teuthology version: 1.0.0-9fd7768 2019-12-11T16:13:36.870 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.PvA_8I.tmp -- --verbose --lock --description rgw/verify/{clusters/fixed-2.yaml frontend/civetweb.yaml msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml overrides.yaml proto/http.yaml rgw_pool_type/ec-profile.yaml tasks/{0-install.yaml cls.yaml ragweed.yaml s3tests-java.yaml s3tests.yaml swift.yaml} validater/valgrind.yaml} --name abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi --block --owner scheduled_abhi@teuthology --archive /home/teuthworker/archive/abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi/4593208 2019-12-11T16:13:37.013 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-11T16:13:37.142 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi/4593208 branch: wip-abhi-testing-2019-12-10-1759 description: rgw/verify/{clusters/fixed-2.yaml frontend/civetweb.yaml msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml overrides.yaml proto/http.yaml rgw_pool_type/ec-profile.yaml tasks/{0-install.yaml cls.yaml ragweed.yaml s3tests-java.yaml s3tests.yaml swift.yaml} validater/valgrind.yaml} email: null first_in_suite: false job_id: '4593208' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos overrides: admin_socket: branch: wip-abhi-testing-2019-12-10-1759 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 global: mon client directed command retry: 5 ms inject socket failures: 5000 osd heartbeat grace: 40 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 1b78fd04fb1980223b57f0a3f644592e4e26b518 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: notcmalloc sha1: 1b78fd04fb1980223b57f0a3f644592e4e26b518 rgw: compression type: random ec-data-pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: testprofile frontend: civetweb storage classes: LUKEWARM, FROZEN s3tests: slow_backend: true thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-abhi-testing-2019-12-10-1759 sha1: 1b78fd04fb1980223b57f0a3f644592e4e26b518 owner: scheduled_abhi@teuthology priority: 99 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: 1b78fd04fb1980223b57f0a3f644592e4e26b518 suite: rgw suite_branch: wip-abhi-testing-2019-12-10-1759 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-10-1759/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 1b78fd04fb1980223b57f0a3f644592e4e26b518 tasks: - install: flavor: notcmalloc - ceph: null - openssl_keys: null - rgw: client.0: valgrind: - --tool=memcheck - --max-threads=1024 - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - rgw/test_rgw_gc_log.sh - ragweed: client.0: default-branch: ceph-master rgw_server: client.0 stages: prepare,check - s3tests-java: client.0: force-branch: master force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: public-access-qa git_remote: git://github.com/theanalyst/ rgw_server: client.0 - swift: client.0: rgw_server: client.0 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.13073 2019-12-11T16:13:37.142 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-10-1759/qa; will attempt to use it 2019-12-11T16:13:37.143 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-10-1759/qa/tasks 2019-12-11T16:13:37.143 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-12-11T16:13:37.156 INFO:teuthology.task.internal:Checking packages... 2019-12-11T16:13:37.209 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'notcmalloc' and ceph hash '1b78fd04fb1980223b57f0a3f644592e4e26b518' 2019-12-11T16:13:37.209 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-12-11T16:13:37.209 INFO:teuthology.packaging:ref: None 2019-12-11T16:13:37.209 INFO:teuthology.packaging:tag: None 2019-12-11T16:13:37.210 INFO:teuthology.packaging:branch: wip-abhi-testing-2019-12-10-1759 2019-12-11T16:13:37.210 INFO:teuthology.packaging:sha1: 1b78fd04fb1980223b57f0a3f644592e4e26b518 2019-12-11T16:13:37.210 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&ref=wip-abhi-testing-2019-12-10-1759 2019-12-11T16:13:37.399 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-7938.g1b78fd0 2019-12-11T16:13:37.399 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-12-11T16:13:37.410 INFO:teuthology.task.internal:no buildpackages task found 2019-12-11T16:13:37.411 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-12-11T16:13:37.426 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-12-11T16:13:37.427 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-11T16:13:37.489 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-11T16:13:47.506 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-11T16:13:57.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-11T16:14:07.572 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-11T16:14:17.589 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-11T16:14:27.611 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_abhi@teuthology', 'description': '/home/teuthworker/archive/abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi/4593208', 'machine_type': 'smithi'} 2019-12-11T16:14:27.645 DEBUG:teuthology.lock.ops:locked ubuntu@smithi079.front.sepia.ceph.com, ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:14:27.756 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-11T16:14:27.992 INFO:teuthology.provision.fog.smithi199:Scheduling deploy of centos 7.6 2019-12-11T16:14:27.995 INFO:teuthology.provision.fog.smithi079:Scheduling deploy of centos 7.6 2019-12-11T16:14:28.516 INFO:teuthology.orchestra.console:Power off smithi199 2019-12-11T16:14:28.516 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-11T16:14:28.539 INFO:teuthology.orchestra.console:Power off smithi079 2019-12-11T16:14:28.539 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-11T16:14:28.550 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:28.572 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:32.584 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:32.599 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:36.700 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:36.717 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:36.915 INFO:teuthology.orchestra.console:Power off for smithi199 completed 2019-12-11T16:14:37.016 INFO:teuthology.orchestra.console:Power on smithi199 2019-12-11T16:14:37.016 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-11T16:14:37.045 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:40.833 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:41.048 INFO:teuthology.orchestra.console:Power off for smithi079 completed 2019-12-11T16:14:41.082 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:41.150 INFO:teuthology.orchestra.console:Power on smithi079 2019-12-11T16:14:41.151 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-11T16:14:41.175 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:45.200 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:45.211 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:45.417 INFO:teuthology.orchestra.console:Power on for smithi199 completed 2019-12-11T16:14:45.518 INFO:teuthology.provision.fog.smithi199:Waiting for deploy to finish 2019-12-11T16:14:49.332 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-11T16:14:49.547 INFO:teuthology.orchestra.console:Power on for smithi079 completed 2019-12-11T16:14:49.648 INFO:teuthology.provision.fog.smithi079:Waiting for deploy to finish 2019-12-11T16:17:16.950 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:17:26.033 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:17:35.128 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:17:35.986 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:17:44.206 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:17:45.069 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:17:53.261 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:17:54.157 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:02.321 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:03.209 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:11.405 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:12.266 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:21.325 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:30.381 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:39.437 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:18:43.287 INFO:teuthology.orchestra.run.smithi199:> hostname 2019-12-11T16:18:43.753 INFO:teuthology.orchestra.run.smithi199.stdout:smithi068 2019-12-11T16:18:43.754 INFO:teuthology.orchestra.run.smithi199:> grep smithi068 /etc/hosts 2019-12-11T16:18:43.808 INFO:teuthology.orchestra.run.smithi199.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2019-12-11T16:18:43.808 INFO:teuthology.orchestra.run.smithi199:> sudo hostname smithi199 2019-12-11T16:18:43.940 INFO:teuthology.orchestra.run.smithi199:> sudo sed -i -e 's/smithi068/smithi199/g' /etc/hosts 2019-12-11T16:18:44.045 INFO:teuthology.orchestra.run.smithi199:> sudo sed -i -e 's/172.21.15.68/172.21.15.199/g' /etc/hosts 2019-12-11T16:18:44.125 INFO:teuthology.orchestra.run.smithi199:> sudo sed -i -e 's/smithi068/smithi199/g' /etc/hostname 2019-12-11T16:18:44.193 INFO:teuthology.orchestra.run.smithi199:> sudo hostname smithi199 2019-12-11T16:18:44.254 INFO:teuthology.provision.fog.smithi199:Deploy complete! 2019-12-11T16:18:44.256 DEBUG:teuthology.parallel:result is None 2019-12-11T16:18:48.493 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:19:54.499 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:19:54.694 INFO:teuthology.orchestra.run.smithi079:> hostname 2019-12-11T16:19:55.280 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 2019-12-11T16:19:55.280 INFO:teuthology.orchestra.run.smithi079:> grep smithi079 /etc/hosts 2019-12-11T16:19:55.366 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79 smithi079.front.sepia.ceph.com smithi079 2019-12-11T16:19:55.366 INFO:teuthology.orchestra.run.smithi079:> sudo hostname smithi079 2019-12-11T16:19:55.523 INFO:teuthology.orchestra.run.smithi079:> sudo sed -i -e 's/smithi079/smithi079/g' /etc/hosts 2019-12-11T16:19:55.638 INFO:teuthology.orchestra.run.smithi079:> sudo sed -i -e 's/172.21.15.79/172.21.15.79/g' /etc/hosts 2019-12-11T16:19:55.741 INFO:teuthology.orchestra.run.smithi079:> sudo sed -i -e 's/smithi079/smithi079/g' /etc/hostname 2019-12-11T16:19:55.830 INFO:teuthology.orchestra.run.smithi079:> sudo hostname smithi079 2019-12-11T16:19:55.903 INFO:teuthology.provision.fog.smithi079:Deploy complete! 2019-12-11T16:19:55.910 DEBUG:teuthology.parallel:result is None 2019-12-11T16:19:56.075 INFO:teuthology.lock.ops:Checking smithi079.front.sepia.ceph.com 2019-12-11T16:19:56.076 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-11T16:19:56.094 INFO:teuthology.lock.ops:Checking smithi199.front.sepia.ceph.com 2019-12-11T16:19:56.094 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-11T16:19:56.109 INFO:teuthology.lock.ops:Updating [smithi079.front.sepia.ceph.com]: set os type and version on server 2019-12-11T16:19:56.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:19:56.110 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:19:56.189 INFO:teuthology.orchestra.run.smithi079:> true 2019-12-11T16:19:56.581 INFO:teuthology.orchestra.run.smithi079:> uname -m 2019-12-11T16:19:56.649 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2019-12-11T16:19:56.650 INFO:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2019-12-11T16:19:56.718 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Linux" 2019-12-11T16:19:56.718 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="7 (Core)" 2019-12-11T16:19:56.718 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2019-12-11T16:19:56.718 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2019-12-11T16:19:56.720 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="7" 2019-12-11T16:19:56.720 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-11T16:19:56.720 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2019-12-11T16:19:56.720 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-11T16:19:56.721 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://www.centos.org/" 2019-12-11T16:19:56.721 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-11T16:19:56.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:19:56.721 INFO:teuthology.orchestra.run.smithi079.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-11T16:19:56.721 INFO:teuthology.orchestra.run.smithi079.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-11T16:19:56.721 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-11T16:19:56.721 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-11T16:19:56.722 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:19:56.722 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2019-12-11T16:19:56.736 INFO:teuthology.lock.ops:Updating [smithi199.front.sepia.ceph.com]: set os type and version on server 2019-12-11T16:19:56.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:19:56.736 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:19:56.827 INFO:teuthology.orchestra.run.smithi199:> true 2019-12-11T16:19:57.223 INFO:teuthology.orchestra.run.smithi199:> uname -m 2019-12-11T16:19:57.307 INFO:teuthology.orchestra.run.smithi199.stdout:x86_64 2019-12-11T16:19:57.308 INFO:teuthology.orchestra.run.smithi199:> cat /etc/os-release 2019-12-11T16:19:57.390 INFO:teuthology.orchestra.run.smithi199.stdout:NAME="CentOS Linux" 2019-12-11T16:19:57.390 INFO:teuthology.orchestra.run.smithi199.stdout:VERSION="7 (Core)" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:ID="centos" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:ID_LIKE="rhel fedora" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:VERSION_ID="7" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:ANSI_COLOR="0;31" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:HOME_URL="https://www.centos.org/" 2019-12-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi199.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-11T16:19:57.393 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:19:57.393 INFO:teuthology.orchestra.run.smithi199.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-11T16:19:57.393 INFO:teuthology.orchestra.run.smithi199.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-11T16:19:57.393 INFO:teuthology.orchestra.run.smithi199.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-11T16:19:57.393 INFO:teuthology.orchestra.run.smithi199.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-11T16:19:57.393 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:19:57.395 INFO:teuthology.lock.ops:Updating smithi199.front.sepia.ceph.com on lock server 2019-12-11T16:19:57.418 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-12-11T16:19:57.453 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi079.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZGRxxko5Y8zu5VAwr2MZuVC5fMBMybUCjvady+TahEihj8hVnQvx5rPKmqqklR6f2c7CYzISj2jSHle4wqQVgT4mkjLGEPZq5EXE4cHJSkjk2ngXOD5mD0ehhF8XQq+3zhhLY2P/JSw+aEBejmT4gxcrv9o98nwjMWEuT5Ks7o2q0Oi9Xaiy4GNtV0ST4BvUOx5qMQi0c6Uxz9k2y+rEk+9OEPnbbgyQlzYPsD0pHfBulfcBVkx2GLjgSrqDV1k7p6dKwj2vGSCNWDi+ve7rSfLo4bQvUekuKA+F2V987PwIlJhn/w8E5qrFm9ocq1R/u/+Y8O1kjcFBVZnM00E6r smithi199.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRzwD4606ZRlVlNgyRMRmn6nTnSbBQDP9absvlSeLbl1jXur1CG+cMVJQ4AgfWt7lz63z1hFkAFkbETWRJJNNI3X8ML8g/wbR1Myxw+a164PNCqoR+jswwdyh6fGID3R1In9ckQgRIhzQMANFbRKy2fkqB6yqhp2F6tI75hCwBEKF5OKWb7XTxmEe5qdyRMgJmCrObCPuyOQTiJWbypBGKOGMfeTfdBbYaO/Xo6fJESoIVnkiUwBaTZvrqtDTo720WJJjYh0JeMhnFYnB5p4OcLCZaz3wUgwhl2lz8c6yhIdacEV6cb0apwCvBfmGXWUJGMYpjYqWW0mXR7RlCNNhf 2019-12-11T16:19:57.453 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-11T16:19:57.498 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-12-11T16:19:57.504 INFO:teuthology.task.internal:Saving configuration 2019-12-11T16:19:57.523 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-12-11T16:19:57.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-12-11T16:19:57.546 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-11 16:14:27.628150', u'locked_by': u'scheduled_abhi@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi079.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCwvjlNUSym8py0bC+QV+eyBF9bqPBWZKeXG6yWRpEcOAQ/AogfcTQECfF/n3qeGZ9E9Mv0w/s4bVGBwcPNOJos=', u'description': u'/home/teuthworker/archive/abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi/4593208'} 2019-12-11T16:19:57.561 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-11 16:14:27.628131', u'locked_by': u'scheduled_abhi@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi199.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZRhbHupwmxZ15hkO6wRUjGt/YioDDYUcufcy8XIrqf7GzEOEHk0y8BRNzabnrRX176bbd5+dZtXLl+3hLR+aE=', u'description': u'/home/teuthworker/archive/abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi/4593208'} 2019-12-11T16:19:57.561 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-12-11T16:19:57.570 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-12-11T16:19:57.570 INFO:teuthology.task.internal:roles: ubuntu@smithi199.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2019-12-11T16:19:57.570 INFO:teuthology.run_tasks:Running task console_log... 2019-12-11T16:19:57.666 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-11T16:19:57.668 INFO:teuthology.run_tasks:Running task internal.connect... 2019-12-11T16:19:57.676 INFO:teuthology.task.internal:Opening connections... 2019-12-11T16:19:57.676 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:19:57.678 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:19:57.774 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:19:57.775 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:19:57.872 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-12-11T16:19:57.887 INFO:teuthology.orchestra.run.smithi079:> uname -m 2019-12-11T16:19:58.084 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2019-12-11T16:19:58.085 INFO:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2019-12-11T16:19:58.141 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Linux" 2019-12-11T16:19:58.142 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="7 (Core)" 2019-12-11T16:19:58.142 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2019-12-11T16:19:58.142 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2019-12-11T16:19:58.143 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="7" 2019-12-11T16:19:58.143 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-11T16:19:58.143 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2019-12-11T16:19:58.143 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-11T16:19:58.143 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://www.centos.org/" 2019-12-11T16:19:58.144 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-11T16:19:58.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:19:58.147 INFO:teuthology.orchestra.run.smithi079.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-11T16:19:58.147 INFO:teuthology.orchestra.run.smithi079.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-11T16:19:58.147 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-11T16:19:58.147 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-11T16:19:58.148 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:19:58.148 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2019-12-11T16:19:58.168 INFO:teuthology.orchestra.run.smithi199:> uname -m 2019-12-11T16:19:58.288 INFO:teuthology.orchestra.run.smithi199.stdout:x86_64 2019-12-11T16:19:58.288 INFO:teuthology.orchestra.run.smithi199:> cat /etc/os-release 2019-12-11T16:19:58.357 INFO:teuthology.orchestra.run.smithi199.stdout:NAME="CentOS Linux" 2019-12-11T16:19:58.358 INFO:teuthology.orchestra.run.smithi199.stdout:VERSION="7 (Core)" 2019-12-11T16:19:58.358 INFO:teuthology.orchestra.run.smithi199.stdout:ID="centos" 2019-12-11T16:19:58.358 INFO:teuthology.orchestra.run.smithi199.stdout:ID_LIKE="rhel fedora" 2019-12-11T16:19:58.358 INFO:teuthology.orchestra.run.smithi199.stdout:VERSION_ID="7" 2019-12-11T16:19:58.359 INFO:teuthology.orchestra.run.smithi199.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-11T16:19:58.359 INFO:teuthology.orchestra.run.smithi199.stdout:ANSI_COLOR="0;31" 2019-12-11T16:19:58.359 INFO:teuthology.orchestra.run.smithi199.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-11T16:19:58.359 INFO:teuthology.orchestra.run.smithi199.stdout:HOME_URL="https://www.centos.org/" 2019-12-11T16:19:58.360 INFO:teuthology.orchestra.run.smithi199.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-11T16:19:58.360 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:19:58.360 INFO:teuthology.orchestra.run.smithi199.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-11T16:19:58.361 INFO:teuthology.orchestra.run.smithi199.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-11T16:19:58.361 INFO:teuthology.orchestra.run.smithi199.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-11T16:19:58.361 INFO:teuthology.orchestra.run.smithi199.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-11T16:19:58.361 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:19:58.362 INFO:teuthology.lock.ops:Updating smithi199.front.sepia.ceph.com on lock server 2019-12-11T16:19:58.373 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-12-11T16:19:58.397 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-12-11T16:19:58.406 INFO:teuthology.task.internal:Checking for old test directory... 2019-12-11T16:19:58.406 INFO:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2019-12-11T16:19:58.410 INFO:teuthology.orchestra.run.smithi199:> test '!' -e /home/ubuntu/cephtest 2019-12-11T16:19:58.510 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-12-11T16:19:58.519 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-12-11T16:19:58.519 INFO:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2019-12-11T16:19:58.523 INFO:teuthology.orchestra.run.smithi199:> test -z $(ls -A /var/lib/ceph) 2019-12-11T16:19:58.629 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-12-11T16:19:58.674 INFO:teuthology.run_tasks:Running task kernel... 2019-12-11T16:19:58.700 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}} 2019-12-11T16:19:58.700 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-12-11T16:19:58.700 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:19:58.701 INFO:teuthology.orchestra.run.smithi079:> sudo package-cleanup -y --oldkernels 2019-12-11T16:19:59.638 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:20:01.250 INFO:teuthology.orchestra.run.smithi079.stdout:No old kernels to remove 2019-12-11T16:20:01.266 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:20:01.267 INFO:teuthology.orchestra.run.smithi199:> sudo package-cleanup -y --oldkernels 2019-12-11T16:20:02.201 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:20:04.233 INFO:teuthology.orchestra.run.smithi199.stdout:No old kernels to remove 2019-12-11T16:20:04.234 INFO:teuthology.orchestra.run.smithi079:> uname -r 2019-12-11T16:20:04.353 INFO:teuthology.orchestra.run.smithi079.stdout:3.10.0-1062.4.1.el7.x86_64 2019-12-11T16:20:04.354 INFO:teuthology.task.kernel:Running kernel on smithi079: 3.10.0-1062.4.1.el7.x86_64 2019-12-11T16:20:04.354 INFO:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2019-12-11T16:20:04.647 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:20:04.718 INFO:teuthology.orchestra.run.smithi079.stdout:Determining fastest mirrors 2019-12-11T16:20:04.925 INFO:teuthology.orchestra.run.smithi079.stdout: * base: centos.mirror.constant.com 2019-12-11T16:20:04.925 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: ftp.linux.ncsu.edu 2019-12-11T16:20:04.926 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.atlanticmetro.net 2019-12-11T16:20:04.927 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:20:07.071 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:20:07.071 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:20:07.072 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package kernel.x86_64 0:3.10.0-1062.9.1.el7 will be installed 2019-12-11T16:20:08.353 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:20:09.111 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:20:09.111 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:20:09.112 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:20:09.112 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:20:09.112 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:20:09.112 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:20:09.112 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:20:09.113 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 3.10.0-1062.9.1.el7 updates 50 M 2019-12-11T16:20:09.113 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:20:09.113 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:20:09.113 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:20:09.113 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2019-12-11T16:20:09.113 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:20:09.114 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 50 M 2019-12-11T16:20:09.114 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 64 M 2019-12-11T16:20:09.114 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:20:09.114 INFO:teuthology.orchestra.run.smithi079.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-11T16:20:10.173 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:20:10.367 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:20:11.279 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:20:11.279 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:20:46.312 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-11T16:20:46.821 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-11T16:20:46.821 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:20:46.821 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:20:46.821 INFO:teuthology.orchestra.run.smithi079.stdout: kernel.x86_64 0:3.10.0-1062.9.1.el7 2019-12-11T16:20:46.821 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:20:46.822 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:20:46.874 INFO:teuthology.orchestra.run.smithi079:> rpm -q kernel --last | head -n 1 2019-12-11T16:20:47.265 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Wed 11 Dec 2019 04:20:14 PM UTC 2019-12-11T16:20:47.266 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:20:47.267 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:20:47.267 INFO:teuthology.orchestra.run.smithi199:> uname -r 2019-12-11T16:20:47.375 INFO:teuthology.orchestra.run.smithi199.stdout:3.10.0-1062.4.1.el7.x86_64 2019-12-11T16:20:47.376 INFO:teuthology.task.kernel:Running kernel on smithi199: 3.10.0-1062.4.1.el7.x86_64 2019-12-11T16:20:47.376 INFO:teuthology.orchestra.run.smithi199:> sudo yum install -y kernel 2019-12-11T16:20:47.656 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:20:47.693 INFO:teuthology.orchestra.run.smithi199.stdout:Determining fastest mirrors 2019-12-11T16:20:47.900 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:20:47.901 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: ftp.linux.ncsu.edu 2019-12-11T16:20:47.901 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:20:47.902 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: mirror.ash.fastserv.com 2019-12-11T16:20:50.370 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:20:50.370 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:20:50.371 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package kernel.x86_64 0:3.10.0-1062.9.1.el7 will be installed 2019-12-11T16:20:51.661 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:20:52.499 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:20:52.500 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:20:52.527 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:20:52.527 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:20:52.527 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository Size 2019-12-11T16:20:52.527 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:20:52.528 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:20:52.528 INFO:teuthology.orchestra.run.smithi199.stdout: kernel x86_64 3.10.0-1062.9.1.el7 updates 50 M 2019-12-11T16:20:52.528 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:20:52.528 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:20:52.528 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:20:52.528 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package 2019-12-11T16:20:52.530 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:20:52.530 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 50 M 2019-12-11T16:20:52.530 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 64 M 2019-12-11T16:20:52.530 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:20:52.545 INFO:teuthology.orchestra.run.smithi199.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-11T16:20:53.807 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:20:53.985 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:20:54.875 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:20:54.876 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:21:29.179 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-11T16:21:29.754 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-11T16:21:29.754 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:21:29.754 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:21:29.754 INFO:teuthology.orchestra.run.smithi199.stdout: kernel.x86_64 0:3.10.0-1062.9.1.el7 2019-12-11T16:21:29.755 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:21:29.755 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:21:29.809 INFO:teuthology.orchestra.run.smithi199:> rpm -q kernel --last | head -n 1 2019-12-11T16:21:30.198 INFO:teuthology.orchestra.run.smithi199.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Wed 11 Dec 2019 04:20:57 PM UTC 2019-12-11T16:21:30.200 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:30.200 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:30.200 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-12-11T16:21:30.200 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-11T16:21:30.200 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-11T16:21:30.200 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-12-11T16:21:30.201 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2019-12-11T16:21:30.201 INFO:teuthology.orchestra.run.smithi079:> rpm -q kernel --last | head -n 1 2019-12-11T16:21:30.567 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Wed 11 Dec 2019 04:20:14 PM UTC 2019-12-11T16:21:30.569 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:30.570 INFO:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2 2019-12-11T16:21:30.734 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2 2019-12-11T16:21:30.734 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2019-12-11T16:21:30.734 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.02 2019-12-11T16:21:30.735 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 0.76.el7.centos 2019-12-11T16:21:30.735 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2019-12-11T16:21:30.735 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Tue 05 Nov 2019 03:41:14 PM UTC 2019-12-11T16:21:30.735 INFO:teuthology.orchestra.run.smithi079.stdout:Group : System Environment/Base 2019-12-11T16:21:30.735 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 0 2019-12-11T16:21:30.735 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2019-12-11T16:21:30.735 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-12-11T16:21:30.736 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-12-11T16:21:30.736 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-12-11T16:21:30.736 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : kbuilder.bsys.centos.org 2019-12-11T16:21:30.736 INFO:teuthology.orchestra.run.smithi079.stdout:Relocations : (not relocatable) 2019-12-11T16:21:30.736 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : CentOS BuildSystem 2019-12-11T16:21:30.736 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2019-12-11T16:21:30.736 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2019-12-11T16:21:30.737 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-12-11T16:21:30.737 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2019-12-11T16:21:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:21:30.737 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-12-11T16:21:30.737 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2019-12-11T16:21:30.737 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2019-12-11T16:21:30.737 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2019-12-11T16:21:30.738 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-12-11T16:21:30.738 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:30.739 INFO:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-12-11T16:21:31.218 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2019-12-11T16:21:31.360 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:31.375 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.9.1.el7.x86_64.img 2019-12-11T16:21:31.656 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-12-11T16:21:31.693 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-12-11T16:21:31.749 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-12-11T16:21:31.753 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-12-11T16:21:31.783 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-12-11T16:21:31.795 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-12-11T16:21:32.781 INFO:teuthology.orchestra.run.smithi079.stderr:done 2019-12-11T16:21:32.782 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:21:32.841 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.F830FkfE7N 2019-12-11T16:21:32.841 INFO:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.F830FkfE7N 2019-12-11T16:21:32.907 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.F830FkfE7N 2019-12-11T16:21:33.024 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.F830FkfE7N is 6KB 2019-12-11T16:21:33.093 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.F830FkfE7N 2019-12-11T16:21:33.165 INFO:teuthology.orchestra.run.smithi079:> sudo grub2-set-default 0 2019-12-11T16:21:33.413 INFO:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2019-12-11T16:21:33.417 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-12-11T16:21:33.418 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi199.front.sepia.ceph.com, path=None, version=distro) 2019-12-11T16:21:33.418 INFO:teuthology.orchestra.run.smithi199:> rpm -q kernel --last | head -n 1 2019-12-11T16:21:33.783 INFO:teuthology.orchestra.run.smithi199.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Wed 11 Dec 2019 04:20:57 PM UTC 2019-12-11T16:21:33.800 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:33.801 INFO:teuthology.orchestra.run.smithi199:> sudo rpm -qi grub2 2019-12-11T16:21:33.937 INFO:teuthology.orchestra.run.smithi199.stdout:Name : grub2 2019-12-11T16:21:33.938 INFO:teuthology.orchestra.run.smithi199.stdout:Epoch : 1 2019-12-11T16:21:33.939 INFO:teuthology.orchestra.run.smithi199.stdout:Version : 2.02 2019-12-11T16:21:33.939 INFO:teuthology.orchestra.run.smithi199.stdout:Release : 0.76.el7.centos 2019-12-11T16:21:33.940 INFO:teuthology.orchestra.run.smithi199.stdout:Architecture: x86_64 2019-12-11T16:21:33.940 INFO:teuthology.orchestra.run.smithi199.stdout:Install Date: Tue 05 Nov 2019 03:41:14 PM UTC 2019-12-11T16:21:33.940 INFO:teuthology.orchestra.run.smithi199.stdout:Group : System Environment/Base 2019-12-11T16:21:33.940 INFO:teuthology.orchestra.run.smithi199.stdout:Size : 0 2019-12-11T16:21:33.940 INFO:teuthology.orchestra.run.smithi199.stdout:License : GPLv3+ 2019-12-11T16:21:33.940 INFO:teuthology.orchestra.run.smithi199.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-12-11T16:21:33.940 INFO:teuthology.orchestra.run.smithi199.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-12-11T16:21:33.941 INFO:teuthology.orchestra.run.smithi199.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-12-11T16:21:33.941 INFO:teuthology.orchestra.run.smithi199.stdout:Build Host : kbuilder.bsys.centos.org 2019-12-11T16:21:33.941 INFO:teuthology.orchestra.run.smithi199.stdout:Relocations : (not relocatable) 2019-12-11T16:21:33.941 INFO:teuthology.orchestra.run.smithi199.stdout:Packager : CentOS BuildSystem 2019-12-11T16:21:33.941 INFO:teuthology.orchestra.run.smithi199.stdout:Vendor : CentOS 2019-12-11T16:21:33.941 INFO:teuthology.orchestra.run.smithi199.stdout:URL : http://www.gnu.org/software/grub/ 2019-12-11T16:21:33.941 INFO:teuthology.orchestra.run.smithi199.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-12-11T16:21:33.942 INFO:teuthology.orchestra.run.smithi199.stdout:Description : 2019-12-11T16:21:33.942 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:21:33.942 INFO:teuthology.orchestra.run.smithi199.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-12-11T16:21:33.943 INFO:teuthology.orchestra.run.smithi199.stdout:customizable bootloader with modular architecture. It supports a rich 2019-12-11T16:21:33.943 INFO:teuthology.orchestra.run.smithi199.stdout:variety of kernel formats, file systems, computer architectures and 2019-12-11T16:21:33.943 INFO:teuthology.orchestra.run.smithi199.stdout:hardware devices. 2019-12-11T16:21:33.944 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-12-11T16:21:33.944 INFO:teuthology.task.kernel:Updating grub on smithi199 to boot 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:33.944 INFO:teuthology.orchestra.run.smithi199:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-12-11T16:21:34.443 INFO:teuthology.orchestra.run.smithi199.stderr:Generating grub configuration file ... 2019-12-11T16:21:34.574 INFO:teuthology.orchestra.run.smithi199.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:21:34.589 INFO:teuthology.orchestra.run.smithi199.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.9.1.el7.x86_64.img 2019-12-11T16:21:34.847 INFO:teuthology.orchestra.run.smithi199.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-12-11T16:21:34.867 INFO:teuthology.orchestra.run.smithi199.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-12-11T16:21:34.902 INFO:teuthology.orchestra.run.smithi199.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-12-11T16:21:34.919 INFO:teuthology.orchestra.run.smithi199.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-12-11T16:21:34.947 INFO:teuthology.orchestra.run.smithi199.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-12-11T16:21:34.957 INFO:teuthology.orchestra.run.smithi199.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-12-11T16:21:36.088 INFO:teuthology.orchestra.run.smithi199.stderr:done 2019-12-11T16:21:36.089 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:21:36.223 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.EsDUVfR5f9 2019-12-11T16:21:36.223 INFO:teuthology.orchestra.run.smithi199:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EsDUVfR5f9 2019-12-11T16:21:36.334 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.EsDUVfR5f9 2019-12-11T16:21:36.544 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.EsDUVfR5f9 is 6KB 2019-12-11T16:21:36.639 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.EsDUVfR5f9 2019-12-11T16:21:36.726 INFO:teuthology.orchestra.run.smithi199:> sudo grub2-set-default 0 2019-12-11T16:21:36.939 INFO:teuthology.orchestra.run.smithi199:> sudo shutdown -r now 2019-12-11T16:21:36.944 INFO:teuthology.misc:Re-opening connections... 2019-12-11T16:21:36.945 INFO:teuthology.misc:trying to connect to ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:21:36.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:21:36.945 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:22:36.947 DEBUG:teuthology.orchestra.remote:timed out 2019-12-11T16:22:36.947 INFO:teuthology.misc:trying to connect to ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:22:36.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:22:36.948 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:23:36.949 DEBUG:teuthology.orchestra.remote:timed out 2019-12-11T16:23:36.949 DEBUG:teuthology.misc:waited 120.004740953 2019-12-11T16:23:37.959 INFO:teuthology.misc:trying to connect to ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:23:37.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:23:37.960 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:23:38.073 INFO:teuthology.orchestra.run.smithi079:> true 2019-12-11T16:23:38.403 DEBUG:teuthology.misc:waited 121.458199978 2019-12-11T16:23:39.404 INFO:teuthology.misc:trying to connect to ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:23:39.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:23:39.405 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:23:39.518 INFO:teuthology.orchestra.run.smithi199:> true 2019-12-11T16:23:39.934 DEBUG:teuthology.misc:waited 122.989196062 2019-12-11T16:23:40.940 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-12-11T16:23:40.941 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.9.1.el7.x86_64"... 2019-12-11T16:23:40.941 INFO:teuthology.orchestra.run.smithi079:> uname -r 2019-12-11T16:23:41.046 INFO:teuthology.orchestra.run.smithi079.stdout:3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:23:41.046 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:23:41.046 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-11T16:23:41.048 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-12-11T16:23:41.048 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "3.10.0-1062.9.1.el7.x86_64"... 2019-12-11T16:23:41.048 INFO:teuthology.orchestra.run.smithi199:> uname -r 2019-12-11T16:23:41.150 INFO:teuthology.orchestra.run.smithi199.stdout:3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:23:41.151 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.9.1.el7.x86_64 2019-12-11T16:23:41.151 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-11T16:23:42.153 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-12-11T16:23:42.153 INFO:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-11T16:23:42.236 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2019-12-11T16:23:42.261 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-12-11T16:23:42.261 INFO:teuthology.orchestra.run.smithi199:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-11T16:23:42.423 INFO:teuthology.orchestra.run.smithi199.stdout:ttyS1 2019-12-11T16:23:42.439 INFO:teuthology.run_tasks:Running task internal.base... 2019-12-11T16:23:42.456 INFO:teuthology.task.internal:Creating test directory... 2019-12-11T16:23:42.456 INFO:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-11T16:23:42.459 INFO:teuthology.orchestra.run.smithi199:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-11T16:23:42.551 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-12-11T16:23:42.617 INFO:teuthology.run_tasks:Running task internal.archive... 2019-12-11T16:23:42.634 INFO:teuthology.task.internal:Creating archive directory... 2019-12-11T16:23:42.634 INFO:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-11T16:23:42.639 INFO:teuthology.orchestra.run.smithi199:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-11T16:23:42.875 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-12-11T16:23:42.904 INFO:teuthology.task.internal:Enabling coredump saving... 2019-12-11T16:23:42.904 INFO:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-11T16:23:42.910 INFO:teuthology.orchestra.run.smithi199:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-11T16:23:42.990 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-11T16:23:43.029 INFO:teuthology.orchestra.run.smithi199.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-11T16:23:43.031 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-12-11T16:23:43.041 INFO:teuthology.task.internal:Configuring sudo... 2019-12-11T16:23:43.041 INFO:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-11T16:23:43.046 INFO:teuthology.orchestra.run.smithi199:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-11T16:23:43.151 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-12-11T16:23:43.166 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-12-11T16:23:43.166 INFO:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-11T16:23:43.173 INFO:teuthology.orchestra.run.smithi199:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-11T16:23:43.245 INFO:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-11T16:23:43.327 INFO:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-11T16:23:43.413 INFO:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-11T16:23:43.502 INFO:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-11T16:23:43.604 INFO:teuthology.orchestra.run.smithi079:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-11T16:23:43.689 INFO:teuthology.orchestra.run.smithi199:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-11T16:23:43.798 INFO:teuthology.orchestra.run.smithi199:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-11T16:23:43.916 INFO:teuthology.orchestra.run.smithi199:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-11T16:23:44.002 INFO:teuthology.orchestra.run.smithi199:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-11T16:23:44.109 INFO:teuthology.orchestra.run.smithi199:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-11T16:23:44.198 INFO:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2019-12-11T16:23:44.202 INFO:teuthology.orchestra.run.smithi199:> sudo service rsyslog restart 2019-12-11T16:23:44.299 INFO:teuthology.orchestra.run.smithi199.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-12-11T16:23:44.344 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-12-11T16:23:44.379 INFO:teuthology.run_tasks:Running task internal.timer... 2019-12-11T16:23:44.399 INFO:teuthology.task.internal:Starting timer... 2019-12-11T16:23:44.399 INFO:teuthology.run_tasks:Running task pcp... 2019-12-11T16:23:44.443 INFO:teuthology.run_tasks:Running task selinux... 2019-12-11T16:23:44.462 INFO:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2019-12-11T16:23:44.622 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: [ OK ] 2019-12-11T16:23:44.625 INFO:teuthology.orchestra.run.smithi199:> sudo service auditd rotate 2019-12-11T16:23:44.787 INFO:teuthology.orchestra.run.smithi199.stdout:Rotating logs: [ OK ] 2019-12-11T16:23:44.789 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-12-11T16:23:44.789 INFO:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2019-12-11T16:23:44.879 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2019-12-11T16:23:44.880 INFO:teuthology.orchestra.run.smithi199:> /usr/sbin/getenforce 2019-12-11T16:23:44.978 INFO:teuthology.orchestra.run.smithi199.stdout:Permissive 2019-12-11T16:23:44.979 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi199.front.sepia.ceph.com': 'permissive'} 2019-12-11T16:23:44.979 INFO:teuthology.orchestra.run.smithi079:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-12-11T16:23:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-11T16:23:45.096 INFO:teuthology.orchestra.run.smithi199:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-12-11T16:23:45.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-11T16:23:45.213 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-12-11T16:23:45.213 INFO:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2019-12-11T16:23:45.345 INFO:teuthology.orchestra.run.smithi199:> sudo /usr/sbin/setenforce permissive 2019-12-11T16:23:45.478 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-12-11T16:23:45.489 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2019-12-11T16:23:45.490 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2019-12-11T16:23:45.516 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-12-11T16:23:45.520 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi199.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-12-11T16:23:45.957 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-12-11T16:23:47.360 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2019-12-11T16:23:50.447 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-12-11T16:23:50.466 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:50 +0000 (0:00:00.067) 0:00:00.067 **** 2019-12-11T16:23:50.478 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:50 +0000 (0:00:00.012) 0:00:00.079 **** 2019-12-11T16:23:53.732 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-12-11T16:23:53.733 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] 2019-12-11T16:23:53.784 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] 2019-12-11T16:23:53.889 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:53 +0000 (0:00:03.410) 0:00:03.490 **** 2019-12-11T16:23:53.986 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:53 +0000 (0:00:00.088) 0:00:03.579 **** 2019-12-11T16:23:54.408 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-11T16:23:54.444 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:54 +0000 (0:00:00.465) 0:00:04.044 **** 2019-12-11T16:23:54.451 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-11T16:23:54.491 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:54 +0000 (0:00:00.047) 0:00:04.092 **** 2019-12-11T16:23:55.179 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-12-11T16:23:55.180 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-11T16:23:55.229 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:55 +0000 (0:00:00.729) 0:00:04.822 **** ok: [smithi199.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-11T16:23:55.274 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:55 +0000 (0:00:00.051) 0:00:04.874 **** 2019-12-11T16:23:55.280 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:23:55.320 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:55 +0000 (0:00:00.047) 0:00:04.922 **** 2019-12-11T16:23:55.328 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:23:55.370 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:55 +0000 (0:00:00.044) 0:00:04.966 **** 2019-12-11T16:23:55.768 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-12-11T16:23:55.770 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:23:55.771 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:23:55.809 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:55 +0000 (0:00:00.442) 0:00:05.409 **** 2019-12-11T16:23:55.844 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:55 +0000 (0:00:00.036) 0:00:05.445 **** 2019-12-11T16:23:57.338 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081435.87-273534482256107/source", "state": "file", "uid": 0} 2019-12-11T16:23:57.341 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081435.82-28841933579350/source", "state": "file", "uid": 0} 2019-12-11T16:23:57.384 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:57 +0000 (0:00:01.533) 0:00:06.979 **** 2019-12-11T16:23:57.411 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:23:57 +0000 (0:00:00.033) 0:00:07.012 **** 2019-12-11T16:23:57.944 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi079.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:58.011 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:58.279 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:58.356 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:58.633 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:58.670 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:58.989 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi199.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, " 2019-12-11T16:23:58.991 INFO:teuthology.task.ansible.out:comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:59.320 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:59.336 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:59.648 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:59.690 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:23:59.989 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:24:00.054 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-11T16:24:00.175 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-12-11T16:24:00.192 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-12-11T16:24:00.372 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:00 +0000 (0:00:02.957) 0:00:09.969 **** 2019-12-11T16:24:00.405 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:00 +0000 (0:00:00.037) 0:00:10.007 **** 2019-12-11T16:24:01.292 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-12-11T16:24:01.294 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] 2019-12-11T16:24:01.360 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] 2019-12-11T16:24:01.418 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Wednesday 11 December 2019 16:24:01 +0000 (0:00:01.012) 0:00:11.019 **** 2019-12-11T16:24:01.502 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:01 +0000 (0:00:00.083) 0:00:11.103 **** 2019-12-11T16:24:02.155 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi199.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.351285", "end": "2019-12-11 16:24:02.130147", "rc": 0, "start": "2019-12-11 16:24:01.778862", "stderr": "", "stderr_lines": [], "stdout": "c84d080fc97e482f88d240859322d5765ac90366", "stdout_lines": ["c84d080fc97e482f88d240859322d5765ac90366"]} 2019-12-11T16:24:02.198 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.695) 0:00:11.799 **** 2019-12-11T16:24:02.214 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.408609", "end": "2019-12-11 16:24:02.183806", "rc": 0, "start": "2019-12-11 16:24:01.775197", "stderr": "", "stderr_lines": [], "stdout": "c84d080fc97e482f88d240859322d5765ac90366", "stdout_lines": ["c84d080fc97e482f88d240859322d5765ac90366"]} 2019-12-11T16:24:02.254 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.056) 0:00:11.855 **** 2019-12-11T16:24:02.557 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2019-12-11T16:24:02.559 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.007099", "end": "2019-12-11 16:24:02.527764", "failed_when_result": false, "rc": 0, "start": "2019-12-11 16:24:02.520665", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} 2019-12-11T16:24:02.613 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.353) 0:00:12.209 **** 2019-12-11T16:24:02.616 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.016665", "end": "2019-12-11 16:24:02.546309", "failed_when_result": false, "rc": 0, "start": "2019-12-11 16:24:02.529644", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} 2019-12-11T16:24:02.665 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.057) 0:00:12.266 **** 2019-12-11T16:24:02.677 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:02.717 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.051) 0:00:12.318 **** 2019-12-11T16:24:02.723 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:02.767 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.050) 0:00:12.369 **** 2019-12-11T16:24:02.788 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:02.829 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.046) 0:00:12.415 **** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:02.857 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.043) 0:00:12.459 **** 2019-12-11T16:24:02.864 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-12-11T16:24:02.866 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:02.902 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.044) 0:00:12.503 **** 2019-12-11T16:24:02.908 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:02.963 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:02 +0000 (0:00:00.055) 0:00:12.558 **** TASK [users : Filter the managed_users list] *********************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:03.009 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:03 +0000 (0:00:00.044) 0:00:12.603 **** 2019-12-11T16:24:03.012 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:03.050 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:03 +0000 (0:00:00.048) 0:00:12.651 **** 2019-12-11T16:24:03.058 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:03.104 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:03 +0000 (0:00:00.049) 0:00:12.701 **** 2019-12-11T16:24:03.105 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:03.143 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:03 +0000 (0:00:00.043) 0:00:12.744 **** 2019-12-11T16:24:03.430 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-12-11T16:24:03.431 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:24:03.484 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:03 +0000 (0:00:00.337) 0:00:13.082 **** 2019-12-11T16:24:03.487 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:24:03.525 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:03 +0000 (0:00:00.044) 0:00:13.126 **** 2019-12-11T16:24:03.881 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-11T16:24:03.881 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-11T16:24:03.914 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-11T16:24:04.208 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-11T16:24:04.251 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-11T16:24:04.519 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-11T16:24:04.625 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-11T16:24:04.840 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-11T16:24:04.898 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-11T16:24:05.232 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-11T16:24:05.274 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-11T16:24:05.562 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-11T16:24:05.622 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-11T16:24:05.852 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-11T16:24:05.960 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-11T16:24:06.207 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-11T16:24:06.322 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-11T16:24:06.561 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-11T16:24:06.654 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-11T16:24:06.890 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-11T16:24:07.040 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-11T16:24:07.251 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-11T16:24:07.385 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-11T16:24:07.581 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-11T16:24:07.753 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-11T16:24:07.930 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-11T16:24:08.093 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-11T16:24:08.264 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-11T16:24:08.383 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-11T16:24:08.624 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-11T16:24:08.742 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-11T16:24:08.955 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-11T16:24:09.093 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-11T16:24:09.278 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-11T16:24:09.443 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-11T16:24:09.623 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-11T16:24:09.807 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-11T16:24:09.971 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-11T16:24:10.145 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-11T16:24:10.332 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-11T16:24:10.507 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-11T16:24:10.694 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-11T16:24:10.881 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-11T16:24:11.021 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-11T16:24:11.233 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-11T16:24:11.332 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-11T16:24:11.618 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-11T16:24:11.670 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-11T16:24:11.990 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-11T16:24:12.047 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-11T16:24:12.365 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-11T16:24:12.404 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-11T16:24:12.723 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-11T16:24:12.764 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-11T16:24:13.078 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-11T16:24:13.108 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-11T16:24:13.409 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-11T16:24:13.420 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-11T16:24:13.736 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-11T16:24:13.747 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-11T16:24:14.093 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-11T16:24:14.096 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-11T16:24:14.425 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-11T16:24:14.469 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-11T16:24:14.764 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-11T16:24:14.820 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-11T16:24:15.084 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-11T16:24:15.160 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-11T16:24:15.430 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-11T16:24:15.517 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-11T16:24:15.770 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-11T16:24:15.877 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-11T16:24:16.104 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-11T16:24:16.221 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-11T16:24:16.415 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-11T16:24:16.582 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-11T16:24:16.770 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-11T16:24:16.929 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-11T16:24:17.156 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-11T16:24:17.296 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-11T16:24:17.450 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-11T16:24:17.653 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-11T16:24:17.817 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-11T16:24:17.987 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-11T16:24:18.144 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-11T16:24:18.339 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-11T16:24:18.505 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-11T16:24:18.680 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-11T16:24:18.827 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-11T16:24:19.031 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-11T16:24:19.200 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-11T16:24:19.347 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-11T16:24:19.552 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-11T16:24:19.707 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-11T16:24:19.890 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-11T16:24:20.068 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-11T16:24:20.226 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-11T16:24:20.408 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-11T16:24:20.535 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-11T16:24:20.748 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-11T16:24:20.879 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-11T16:24:21.124 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-11T16:24:21.232 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-11T16:24:21.434 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-11T16:24:21.573 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-11T16:24:21.788 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-11T16:24:21.936 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-11T16:24:22.181 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-11T16:24:22.298 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-11T16:24:22.514 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-11T16:24:22.642 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-11T16:24:22.894 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-11T16:24:22.970 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-11T16:24:23.218 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-11T16:24:23.303 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-11T16:24:23.545 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-11T16:24:23.650 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-11T16:24:23.909 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-11T16:24:24.002 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-11T16:24:24.253 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-11T16:24:24.319 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-11T16:24:24.593 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-11T16:24:24.648 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-11T16:24:24.931 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-11T16:24:25.015 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-11T16:24:25.287 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-11T16:24:25.362 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-11T16:24:25.655 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-11T16:24:25.718 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-11T16:24:25.976 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-11T16:24:26.062 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-11T16:24:26.312 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-11T16:24:26.417 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-11T16:24:26.670 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-11T16:24:26.768 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-11T16:24:26.990 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-11T16:24:27.102 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-11T16:24:27.352 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-11T16:24:27.448 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-11T16:24:27.684 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-11T16:24:27.820 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-11T16:24:28.026 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-11T16:24:28.165 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-11T16:24:28.349 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-11T16:24:28.505 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-11T16:24:28.672 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-11T16:24:28.843 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-11T16:24:29.013 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-11T16:24:29.186 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-11T16:24:29.347 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-11T16:24:29.548 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-11T16:24:29.676 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-11T16:24:29.893 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-11T16:24:29.982 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-11T16:24:30.235 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-11T16:24:30.312 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-11T16:24:30.540 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-11T16:24:30.653 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-11T16:24:30.902 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-11T16:24:30.976 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-11T16:24:31.244 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-11T16:24:31.326 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-11T16:24:31.593 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-11T16:24:31.682 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-11T16:24:31.950 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-11T16:24:32.024 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-11T16:24:32.309 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-11T16:24:32.374 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-11T16:24:32.653 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-11T16:24:32.709 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-11T16:24:32.976 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-11T16:24:33.056 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-11T16:24:33.311 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-11T16:24:33.411 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-11T16:24:33.647 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-11T16:24:33.772 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-11T16:24:33.984 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-11T16:24:34.091 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-11T16:24:34.331 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-11T16:24:34.436 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-11T16:24:34.637 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-11T16:24:34.762 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-11T16:24:34.981 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-11T16:24:35.096 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-11T16:24:35.333 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-11T16:24:35.437 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-11T16:24:35.699 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-11T16:24:35.789 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-11T16:24:36.016 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-11T16:24:36.110 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-11T16:24:36.370 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-11T16:24:36.439 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-11T16:24:36.722 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-11T16:24:36.786 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-11T16:24:37.062 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-11T16:24:37.140 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-11T16:24:37.382 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-11T16:24:37.482 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-11T16:24:37.735 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-11T16:24:37.825 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-11T16:24:38.102 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-11T16:24:38.166 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-11T16:24:38.449 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-11T16:24:38.512 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-11T16:24:38.788 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-11T16:24:38.840 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-11T16:24:39.118 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10102, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 10102} 2019-12-11T16:24:39.169 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-11T16:24:39.492 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10102, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 10102} 2019-12-11T16:24:39.777 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-12-11T16:24:40.120 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-12-11T16:24:40.373 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "name": "toabctl", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-12-11T16:24:40.509 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:40 +0000 (0:00:36.983) 0:00:50.110 **** 2019-12-11T16:24:40.536 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-11T16:24:40.580 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:40 +0000 (0:00:00.070) 0:00:50.181 **** 2019-12-11T16:24:40.631 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-11T16:24:40.640 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-11T16:24:40.642 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-11T16:24:40.643 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-11T16:24:40.645 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-11T16:24:40.646 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJy 2019-12-11T16:24:40.648 INFO:teuthology.task.ansible.out:w 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 2019-12-11T16:24:40.649 INFO:teuthology.task.ansible.out: 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf853 2019-12-11T16:24:40.650 INFO:teuthology.task.ansible.out:2237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36b 2019-12-11T16:24:40.652 INFO:teuthology.task.ansible.out:deb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, 2019-12-11T16:24:40.653 INFO:teuthology.task.ansible.out: {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsL 2019-12-11T16:24:40.654 INFO:teuthology.task.ansible.out:gffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}]}, "changed": false} 2019-12-11T16:24:40.699 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:40 +0000 (0:00:00.119) 0:00:50.300 **** 2019-12-11T16:24:40.716 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-11T16:24:40.722 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "name": "toabctl", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-12-11T16:24:40.890 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:40 +0000 (0:00:00.190) 0:00:50.491 **** 2019-12-11T16:24:40.929 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-11T16:24:40.969 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:40 +0000 (0:00:00.077) 0:00:50.569 **** 2019-12-11T16:24:41.032 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz 2019-12-11T16:24:41.034 INFO:teuthology.task.ansible.out:2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwb 2019-12-11T16:24:41.035 INFO:teuthology.task.ansible.out:x8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db 2019-12-11T16:24:41.037 INFO:teuthology.task.ansible.out:6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239 2019-12-11T16:24:41.039 INFO:teuthology.task.ansible.out:d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", 2019-12-11T16:24:41.040 INFO:teuthology.task.ansible.out: "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com 2019-12-11T16:24:41.042 INFO:teuthology.task.ansible.out:Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 2019-12-11T16:24:41.044 INFO:teuthology.task.ansible.out:6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f8 2019-12-11T16:24:41.046 INFO:teuthology.task.ansible.out:48846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8a 2019-12-11T16:24:41.047 INFO:teuthology.task.ansible.out:a01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalho 2019-12-11T16:24:41.048 INFO:teuthology.task.ansible.out:jr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}]}, "changed": false} 2019-12-11T16:24:41.096 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:41 +0000 (0:00:00.129) 0:00:50.698 **** 2019-12-11T16:24:41.711 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-12-11T16:24:41.719 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com -> localhost] => {"after": "c84d080fc97e482f88d240859322d5765ac90366", "attempts": 1, "before": "c84d080fc97e482f88d240859322d5765ac90366", "changed": false, "remote_url_changed": false} 2019-12-11T16:24:41.784 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:41 +0000 (0:00:00.669) 0:00:51.368 **** 2019-12-11T16:24:41.842 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com -> localhost] => {"after": "c84d080fc97e482f88d240859322d5765ac90366", "attempts": 1, "before": "c84d080fc97e482f88d240859322d5765ac90366", "changed": false, "remote_url_changed": false} 2019-12-11T16:24:41.880 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:24:41 +0000 (0:00:00.113) 0:00:51.481 **** 2019-12-11T16:24:42.184 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-11T16:24:42.185 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-11T16:24:42.268 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-11T16:24:42.497 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-11T16:24:42.606 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-11T16:24:42.850 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-11T16:24:42.976 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-11T16:24:43.209 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-11T16:24:43.316 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-11T16:24:43.600 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-11T16:24:43.602 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-11T16:24:43.690 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-11T16:24:43.692 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-11T16:24:43.948 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-11T16:24:43.991 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-11T16:24:44.315 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-11T16:24:44.317 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-11T16:24:44.337 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-11T16:24:44.339 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-11T16:24:44.646 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-11T16:24:44.678 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-11T16:24:44.992 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-11T16:24:45.003 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.052 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+r 2019-12-11T16:24:45.056 INFO:teuthology.task.ansible.out:R1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.331 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-11T16:24:45.332 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-12-11T16:24:45.342 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.369 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-11T16:24:45.370 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.377 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.379 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.380 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.396 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:24:45.690 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-11T16:24:45.710 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-11T16:24:46.037 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-11T16:24:46.039 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-11T16:24:46.082 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-11T16:24:46.083 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-11T16:24:46.413 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZ 2019-12-11T16:24:46.415 INFO:teuthology.task.ansible.out:m1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-11T16:24:46.768 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-11T16:24:46.780 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-11T16:24:47.119 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-11T16:24:47.132 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-11T16:24:47.460 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-11T16:24:47.498 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-11T16:24:47.797 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-11T16:24:47.852 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-11T16:24:48.140 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-11T16:24:48.215 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-11T16:24:48.490 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-11T16:24:48.492 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-11T16:24:48.595 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-11T16:24:48.597 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-11T16:24:48.875 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-11T16:24:48.876 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-11T16:24:48.961 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-11T16:24:48.963 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-11T16:24:49.220 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-11T16:24:49.325 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-11T16:24:49.605 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-11T16:24:49.681 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-11T16:24:49.986 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-11T16:24:49.989 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-11T16:24:50.026 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-11T16:24:50.028 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-11T16:24:50.348 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-11T16:24:50.381 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-11T16:24:50.684 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-11T16:24:50.728 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-11T16:24:51.035 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-11T16:24:51.069 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-11T16:24:51.373 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-11T16:24:51.429 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-11T16:24:51.733 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0H 2019-12-11T16:24:51.734 INFO:teuthology.task.ansible.out:LysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-11T16:24:51.802 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-11T16:24:52.064 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-11T16:24:52.070 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-11T16:24:52.190 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-11T16:24:52.192 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-11T16:24:52.400 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-11T16:24:52.574 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-11T16:24:52.750 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-11T16:24:52.948 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-11T16:24:53.098 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-11T16:24:53.312 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-11T16:24:53.426 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-11T16:24:53.656 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-11T16:24:53.774 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-11T16:24:54.008 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-11T16:24:54.130 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-11T16:24:54.361 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-11T16:24:54.477 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-11T16:24:54.707 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-11T16:24:54.814 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-11T16:24:55.095 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-11T16:24:55.183 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-11T16:24:55.410 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-11T16:24:55.540 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-11T16:24:55.542 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-11T16:24:55.776 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-11T16:24:55.777 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-11T16:24:55.885 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-11T16:24:55.886 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-11T16:24:56.124 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-11T16:24:56.126 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-11T16:24:56.232 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-11T16:24:56.476 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-11T16:24:56.561 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-11T16:24:56.822 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-11T16:24:56.917 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-11T16:24:57.176 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-11T16:24:57.304 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-11T16:24:57.305 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-11T16:24:57.535 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-11T16:24:57.537 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-11T16:24:57.621 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-11T16:24:57.623 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-11T16:24:57.627 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-11T16:24:57.883 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-11T16:24:57.885 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-11T16:24:57.886 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-11T16:24:57.949 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-11T16:24:58.242 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-11T16:24:58.339 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-11T16:24:58.586 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-11T16:24:58.667 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-11T16:24:58.927 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-11T16:24:58.970 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-11T16:24:59.320 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-11T16:24:59.331 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-11T16:24:59.333 INFO:teuthology.task.ansible.out: 2019-12-11T16:24:59.647 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-11T16:24:59.689 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-11T16:25:00.004 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-11T16:25:00.026 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-11T16:25:00.355 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-11T16:25:00.356 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-11T16:25:00.664 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-11T16:25:00.668 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-11T16:25:00.708 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-11T16:25:01.010 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-11T16:25:01.074 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-11T16:25:01.078 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-11T16:25:01.376 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-11T16:25:01.451 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-11T16:25:01.756 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-11T16:25:01.772 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-11T16:25:01.828 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-11T16:25:02.113 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-11T16:25:02.115 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:02.198 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-11T16:25:02.202 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-11T16:25:02.507 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-11T16:25:02.552 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-11T16:25:02.872 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-11T16:25:02.910 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-11T16:25:03.228 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-11T16:25:03.254 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-11T16:25:03.554 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-11T16:25:03.628 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-11T16:25:03.875 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-11T16:25:03.989 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6H 2019-12-11T16:25:03.991 INFO:teuthology.task.ansible.out:BvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-11T16:25:04.254 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-11T16:25:04.319 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-11T16:25:04.600 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-11T16:25:04.672 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-11T16:25:04.940 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-11T16:25:05.013 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-11T16:25:05.284 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-11T16:25:05.352 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-11T16:25:05.639 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-11T16:25:05.721 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-11T16:25:05.954 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-11T16:25:06.055 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-11T16:25:06.328 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-11T16:25:06.383 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-11T16:25:06.689 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-11T16:25:06.722 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-11T16:25:07.035 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-11T16:25:07.100 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-11T16:25:07.403 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-11T16:25:07.472 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-11T16:25:07.761 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-11T16:25:07.796 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-11T16:25:08.148 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-11T16:25:08.191 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-11T16:25:08.487 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-11T16:25:08.572 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-11T16:25:08.798 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-11T16:25:08.938 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-11T16:25:09.122 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-11T16:25:09.304 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-11T16:25:09.479 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-11T16:25:09.632 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-11T16:25:09.820 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-11T16:25:09.968 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-11T16:25:10.154 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-11T16:25:10.303 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-11T16:25:10.506 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-11T16:25:10.656 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-11T16:25:10.845 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-11T16:25:10.847 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-11T16:25:11.032 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-11T16:25:11.210 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-11T16:25:11.211 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-11T16:25:11.399 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-11T16:25:11.404 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-11T16:25:11.594 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-11T16:25:11.759 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-11T16:25:11.761 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-11T16:25:11.946 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-11T16:25:12.130 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-11T16:25:12.289 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-11T16:25:12.487 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-11T16:25:12.617 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-11T16:25:12.808 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-11T16:25:12.952 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-11T16:25:13.160 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-11T16:25:13.302 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-11T16:25:13.500 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-11T16:25:13.675 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-11T16:25:13.865 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-11T16:25:14.014 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-11T16:25:14.195 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-11T16:25:14.346 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-11T16:25:14.548 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-11T16:25:14.674 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-11T16:25:14.905 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-11T16:25:15.011 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-11T16:25:15.200 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-11T16:25:15.360 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-11T16:25:15.561 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-11T16:25:15.728 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-11T16:25:15.928 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-11T16:25:16.046 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-11T16:25:16.274 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-11T16:25:16.393 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-11T16:25:16.395 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-11T16:25:16.590 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-11T16:25:16.736 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-11T16:25:16.925 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-11T16:25:16.926 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-11T16:25:17.065 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-11T16:25:17.280 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-11T16:25:17.427 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2019-12-11T16:25:17.550 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:17 +0000 (0:00:35.670) 0:01:27.152 **** 2019-12-11T16:25:17.609 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-11T16:25:17.610 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.620 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.632 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.644 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-11T16:25:17.645 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-11T16:25:17.646 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.674 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.676 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.677 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.678 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.689 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:17.938 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2019-12-11T16:25:18.067 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:18 +0000 (0:00:00.516) 0:01:27.668 **** 2019-12-11T16:25:18.075 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-11T16:25:18.080 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf59 2019-12-11T16:25:18.082 INFO:teuthology.task.ansible.out:2607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.125 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.126 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.148 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.150 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.160 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.171 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.182 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.194 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.206 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.395 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-11T16:25:18.578 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-12-11T16:25:18.589 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:18.694 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-11T16:25:18.932 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-11T16:25:19.026 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-11T16:25:19.037 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.048 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.049 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.060 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.071 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.082 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.105 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.117 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.129 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.140 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.151 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.161 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.173 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.185 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.186 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.197 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.209 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.221 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.232 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.244 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.245 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.258 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.267 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.268 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.279 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.290 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.302 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-11T16:25:19.305 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.306 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.318 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.320 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.331 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.342 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.380 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.395 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.413 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.416 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.425 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.427 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.437 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.439 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:19.450 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.460 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.461 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.472 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.484 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.496 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.507 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.508 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.521 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.533 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.543 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.559 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.570 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.584 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.595 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.606 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.618 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.630 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.650 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.653 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.655 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-11T16:25:19.666 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.681 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.683 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.693 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.694 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.707 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.718 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.731 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.742 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.782 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.790 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.791 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.793 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.794 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d2 2019-12-11T16:25:19.796 INFO:teuthology.task.ansible.out:5807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.797 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.799 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:19.799 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.802 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:19.803 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.804 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:19.806 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.807 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:19.808 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.809 INFO:teuthology.task.ansible.out: skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.813 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:19.813 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.815 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.826 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.828 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.840 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.840 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.852 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.852 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.854 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.866 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.867 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.878 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.890 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.902 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.923 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.924 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.935 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.937 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.938 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.940 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:19.944 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.946 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.947 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.966 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:19.966 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.012 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:20 +0000 (0:00:01.944) 0:01:29.613 **** 2019-12-11T16:25:20.021 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.023 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.024 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.025 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.027 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.041 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.043 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.117 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:20 +0000 (0:00:00.105) 0:01:29.718 **** 2019-12-11T16:25:20.125 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-11T16:25:20.126 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.132 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.134 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.136 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.137 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.149 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.161 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.164 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.175 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.188 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.190 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.203 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.215 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.228 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.240 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.257 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.268 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.279 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.292 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.303 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.314 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.314 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.329 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.560 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ans 2019-12-11T16:25:20.562 INFO:teuthology.task.ansible.out:ible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da 2019-12-11T16:25:20.564 INFO:teuthology.task.ansible.out:0"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was 2019-12-11T16:25:20.566 INFO:teuthology.task.ansible.out: False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": 2019-12-11T16:25:20.568 INFO:teuthology.task.ansible.out: "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3 2019-12-11T16:25:20.570 INFO:teuthology.task.ansible.out:819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi199.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", 2019-12-11T16:25:20.571 INFO:teuthology.task.ansible.out:"ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.573 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.575 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:20.585 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.586 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:20.598 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.599 INFO:teuthology.task.ansible.out: skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.612 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.613 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:20.626 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.627 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:20.638 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.640 INFO:teuthology.task.ansible.out: 2019-12-11T16:25:20.705 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:20 +0000 (0:00:00.588) 0:01:30.307 **** 2019-12-11T16:25:20.734 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-12-11T16:25:20.736 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:20.780 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:20 +0000 (0:00:00.074) 0:01:30.381 **** 2019-12-11T16:25:20.838 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-11T16:25:20.840 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-11T16:25:21.120 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-11T16:25:21.164 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-11T16:25:21.451 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-11T16:25:21.473 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-11T16:25:21.790 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-11T16:25:21.801 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-11T16:25:22.104 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-11T16:25:22.159 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-12-11T16:25:22.445 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-11T16:25:22.489 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-12-11T16:25:22.800 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-12-11T16:25:22.840 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-11T16:25:23.126 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-12-11T16:25:23.168 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-11T16:25:23.477 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-11T16:25:23.484 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-11T16:25:23.737 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-11T16:25:23.827 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-11T16:25:24.065 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-11T16:25:24.161 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-11T16:25:24.383 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-11T16:25:24.492 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-11T16:25:24.719 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-11T16:25:24.835 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-11T16:25:25.042 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-11T16:25:25.136 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-11T16:25:25.377 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-11T16:25:25.462 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-11T16:25:25.726 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-11T16:25:25.769 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-11T16:25:26.046 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-11T16:25:26.077 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-11T16:25:26.381 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-11T16:25:26.411 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-11T16:25:26.734 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-11T16:25:26.747 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-11T16:25:27.069 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-11T16:25:27.081 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-11T16:25:27.378 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-11T16:25:27.697 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} changed: [smithi199.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-11T16:25:28.226 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-11T16:25:28.388 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-11T16:25:28.749 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-11T16:25:28.978 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-11T16:25:29.325 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-11T16:25:29.535 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-11T16:25:29.929 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-11T16:25:30.061 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-11T16:25:30.506 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-11T16:25:30.576 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-11T16:25:31.029 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-11T16:25:31.156 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-11T16:25:31.575 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-11T16:25:31.728 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-11T16:25:32.106 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-11T16:25:32.315 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-11T16:25:32.638 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-11T16:25:32.902 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-11T16:25:33.193 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-11T16:25:33.453 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-11T16:25:33.764 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} 2019-12-11T16:25:33.971 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-11T16:25:34.075 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-11T16:25:34.418 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-11T16:25:34.515 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} 2019-12-11T16:25:34.743 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-11T16:25:34.828 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-11T16:25:35.099 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-11T16:25:35.154 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-11T16:25:35.431 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-11T16:25:35.493 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-11T16:25:35.734 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-11T16:25:35.850 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-11T16:25:36.096 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-11T16:25:36.156 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-11T16:25:36.478 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-11T16:25:36.651 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-11T16:25:36.731 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:36 +0000 (0:00:15.950) 0:01:46.332 **** 2019-12-11T16:25:36.813 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-11T16:25:37.418 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-11T16:25:37.494 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:37 +0000 (0:00:00.762) 0:01:47.095 **** 2019-12-11T16:25:37.817 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "checksum": "a4158e68ba2ebc4c7a2b2e8700cb6d232e28e006", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c98770ab2dfd6854e606863577b39ae3", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081536.76-244789541267488/source", "state": "file", "uid": 0} 2019-12-11T16:25:37.855 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:37 +0000 (0:00:00.360) 0:01:47.455 **** 2019-12-11T16:25:37.891 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-11T16:25:37.896 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => { "msg": "Host smithi199.front.sepia.ceph.com is running CentOS 7.6 (Core)" } 2019-12-11T16:25:37.936 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:37 +0000 (0:00:00.082) 0:01:47.538 **** 2019-12-11T16:25:37.976 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-12-11T16:25:37.978 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-11T16:25:38.023 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.087) 0:01:47.625 **** 2019-12-11T16:25:38.052 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-11T16:25:38.054 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:38.101 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.077) 0:01:47.702 **** 2019-12-11T16:25:38.534 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-12-11T16:25:38.535 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.093309", "end": "2019-12-11 16:25:38.504530", "rc": 0, "start": "2019-12-11 16:25:38.411221", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-11T16:25:38.576 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.475) 0:01:48.178 **** 2019-12-11T16:25:38.614 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-11T16:25:38.616 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:38.657 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.081) 0:01:48.259 **** 2019-12-11T16:25:38.663 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2019-12-11T16:25:38.664 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "checksum": "a4158e68ba2ebc4c7a2b2e8700cb6d232e28e006", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c98770ab2dfd6854e606863577b39ae3", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081537.53-148910879117096/source", "state": "file", "uid": 0} 2019-12-11T16:25:38.704 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.046) 0:01:48.305 **** 2019-12-11T16:25:38.709 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-12-11T16:25:38.711 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:38.757 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.053) 0:01:48.359 **** 2019-12-11T16:25:38.763 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-11T16:25:38.764 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => { "msg": "Host smithi079.front.sepia.ceph.com is running CentOS 7.6 (Core)" } 2019-12-11T16:25:38.818 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.061) 0:01:48.420 **** 2019-12-11T16:25:38.825 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-12-11T16:25:38.827 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:38.867 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.048) 0:01:48.469 **** 2019-12-11T16:25:38.873 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-12-11T16:25:38.874 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-11T16:25:38.932 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.053) 0:01:48.522 **** TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:38.975 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:38 +0000 (0:00:00.054) 0:01:48.577 **** 2019-12-11T16:25:38.984 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-11T16:25:38.984 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.024 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.048) 0:01:48.626 **** 2019-12-11T16:25:39.041 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.070 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.045) 0:01:48.671 **** 2019-12-11T16:25:39.120 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-11T16:25:39.122 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.160 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.090) 0:01:48.762 **** 2019-12-11T16:25:39.209 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-12-11T16:25:39.211 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi199.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:39.249 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.088) 0:01:48.850 **** 2019-12-11T16:25:39.278 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.316 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.066) 0:01:48.917 **** 2019-12-11T16:25:39.353 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-12-11T16:25:39.354 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.401 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.080) 0:01:48.998 **** 2019-12-11T16:25:39.433 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-12-11T16:25:39.435 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:39.484 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.087) 0:01:49.085 **** 2019-12-11T16:25:39.489 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-12-11T16:25:39.490 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.106936", "end": "2019-12-11 16:25:39.441134", "rc": 0, "start": "2019-12-11 16:25:39.334198", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-11T16:25:39.532 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.048) 0:01:49.133 **** 2019-12-11T16:25:39.537 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-11T16:25:39.539 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.596 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.063) 0:01:49.197 **** 2019-12-11T16:25:39.601 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-11T16:25:39.603 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.650 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.045) 0:01:49.242 **** TASK [common : Register with subscription-manager.] **************************** skipping: [smithi199.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:39.690 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.048) 0:01:49.291 **** 2019-12-11T16:25:39.697 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-12-11T16:25:39.699 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.734 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.044) 0:01:49.336 **** 2019-12-11T16:25:39.740 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-12-11T16:25:39.742 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.781 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.046) 0:01:49.383 **** 2019-12-11T16:25:39.788 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.829 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.045) 0:01:49.429 **** 2019-12-11T16:25:39.834 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-12-11T16:25:39.836 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.879 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.046) 0:01:49.476 **** 2019-12-11T16:25:39.880 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-12-11T16:25:39.881 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.927 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.045) 0:01:49.522 **** 2019-12-11T16:25:39.929 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:39.969 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:39 +0000 (0:00:00.048) 0:01:49.570 **** 2019-12-11T16:25:39.974 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-12-11T16:25:39.976 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.016 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.048) 0:01:49.618 **** 2019-12-11T16:25:40.022 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-12-11T16:25:40.023 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.065 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.046) 0:01:49.665 **** 2019-12-11T16:25:40.069 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-11T16:25:40.070 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.113 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.046) 0:01:49.711 **** 2019-12-11T16:25:40.115 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-12-11T16:25:40.117 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:40.165 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.055) 0:01:49.766 **** 2019-12-11T16:25:40.181 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi079.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi079.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:40.210 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.045) 0:01:49.812 **** 2019-12-11T16:25:40.216 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-11T16:25:40.217 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.257 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.046) 0:01:49.858 **** 2019-12-11T16:25:40.268 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.301 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.044) 0:01:49.902 **** 2019-12-11T16:25:40.308 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.347 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.046) 0:01:49.948 **** 2019-12-11T16:25:40.352 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.394 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.047) 0:01:49.996 **** 2019-12-11T16:25:40.404 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.439 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.045) 0:01:50.041 **** 2019-12-11T16:25:40.445 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-12-11T16:25:40.446 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:40.483 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.043) 0:01:50.084 **** 2019-12-11T16:25:40.490 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-12-11T16:25:40.492 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.532 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.048) 0:01:50.133 **** 2019-12-11T16:25:40.537 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-11T16:25:40.538 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.577 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.045) 0:01:50.178 **** 2019-12-11T16:25:40.582 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-12-11T16:25:40.583 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.623 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.046) 0:01:50.224 **** 2019-12-11T16:25:40.629 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-12-11T16:25:40.631 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:40.671 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.048) 0:01:50.273 **** 2019-12-11T16:25:40.677 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-12-11T16:25:40.679 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.715 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.043) 0:01:50.316 **** 2019-12-11T16:25:40.720 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-12-11T16:25:40.723 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.761 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.046) 0:01:50.362 **** 2019-12-11T16:25:40.767 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-11T16:25:40.768 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:40.808 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.045) 0:01:50.408 **** 2019-12-11T16:25:40.812 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-12-11T16:25:40.813 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.870 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.062) 0:01:50.471 **** 2019-12-11T16:25:40.875 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-11T16:25:40.923 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.045) 0:01:50.516 **** TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:40.962 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:40 +0000 (0:00:00.046) 0:01:50.562 **** 2019-12-11T16:25:40.968 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-12-11T16:25:40.970 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.013 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.052) 0:01:50.615 **** 2019-12-11T16:25:41.019 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-12-11T16:25:41.020 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.061 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.045) 0:01:50.660 **** 2019-12-11T16:25:41.096 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi079.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:41.134 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.074) 0:01:50.735 **** 2019-12-11T16:25:41.171 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-11T16:25:41.172 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.209 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.075) 0:01:50.811 **** 2019-12-11T16:25:41.260 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.301 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.091) 0:01:50.903 **** 2019-12-11T16:25:41.338 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-12-11T16:25:41.339 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.374 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.072) 0:01:50.975 **** 2019-12-11T16:25:41.380 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-12-11T16:25:41.386 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:25:41.418 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.044) 0:01:51.019 **** 2019-12-11T16:25:41.423 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-12-11T16:25:41.423 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.462 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.044) 0:01:51.064 **** 2019-12-11T16:25:41.489 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-12-11T16:25:41.490 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.534 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.068) 0:01:51.132 **** 2019-12-11T16:25:41.561 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.605 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.073) 0:01:51.205 **** 2019-12-11T16:25:41.633 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-11T16:25:41.635 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-11T16:25:41.677 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.069) 0:01:51.275 **** 2019-12-11T16:25:41.704 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-11T16:25:41.743 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.069) 0:01:51.344 **** 2019-12-11T16:25:41.780 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-12-11T16:25:41.782 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:25:41.820 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:41 +0000 (0:00:00.076) 0:01:51.420 **** 2019-12-11T16:25:42.142 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi079.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:25:42.181 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:42 +0000 (0:00:00.361) 0:01:51.782 **** 2019-12-11T16:25:42.644 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi199.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-12-11T16:25:43.140 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-12-11T16:25:43.564 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-12-11T16:25:43.599 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:43 +0000 (0:00:01.418) 0:01:53.200 **** 2019-12-11T16:25:44.045 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-12-11T16:25:44.091 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:44 +0000 (0:00:00.491) 0:01:53.692 **** 2019-12-11T16:25:44.718 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-12-11T16:25:44.720 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "6467980f3cbb5aa63a3cfe239626f53e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081543.63-46856706628359/source", "state": "file", "uid": 0} 2019-12-11T16:25:45.224 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "6467980f3cbb5aa63a3cfe239626f53e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081544.13-131330190371119/source", "state": "file", "uid": 0} 2019-12-11T16:25:45.735 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "0848bb8607c5b0c5706d8e1930fa7d55", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081544.73-85232245931878/source", "state": "file", "uid": 0} 2019-12-11T16:25:45.774 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:45 +0000 (0:00:01.683) 0:01:55.375 **** 2019-12-11T16:25:46.362 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "0848bb8607c5b0c5706d8e1930fa7d55", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576081545.24-40384836717421/source", "state": "file", "uid": 0} 2019-12-11T16:25:46.370 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-11T16:25:46.371 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made", "warnings": ["Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually"]} 2019-12-11T16:25:46.429 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:46 +0000 (0:00:00.655) 0:01:56.031 **** 2019-12-11T16:25:46.658 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-11T16:25:46.660 INFO:teuthology.task.ansible.out: [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually 2019-12-11T16:25:46.710 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:46 +0000 (0:00:00.280) 0:01:56.312 **** 2019-12-11T16:25:46.763 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made", "warnings": ["Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually"]} 2019-12-11T16:25:47.071 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-11T16:25:47.115 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:25:47 +0000 (0:00:00.404) 0:01:56.716 **** 2019-12-11T16:26:00.314 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** [WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rf /var/cache/yum/*; yum clean all; yum makecache", "delta": "0:00:13.264044", "end": "2019-12-11 16:26:00.286442", "rc": 0, "start": "2019-12-11 16:25:47.022398", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nLoaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.facebook.net\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nMetadata Cache Created", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.facebook.net", " * extras: mirror.facebook.net", " * updates: centos.mirror.constant.com", "Metadata Cache Created"]} 2019-12-11T16:26:00.391 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:00 +0000 (0:00:13.275) 0:02:09.991 **** 2019-12-11T16:26:00.640 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rf /var/cache/yum/*; yum clean all; yum makecache", "delta": "0:00:13.174314", "end": "2019-12-11 16:26:00.599218", "rc": 0, "start": "2019-12-11 16:25:47.424904", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nLoaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.facebook.net\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nMetadata Cache Created", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.facebook.net", " * extras: mirror.facebook.net", " * updates: centos.mirror.constant.com", "Metadata Cache Created"]} 2019-12-11T16:26:00.681 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:00 +0000 (0:00:00.291) 0:02:10.283 **** 2019-12-11T16:26:00.733 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* 2019-12-11T16:26:00.734 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made"} 2019-12-11T16:26:01.041 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made"} 2019-12-11T16:26:01.080 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-11T16:26:01.136 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.453) 0:02:10.736 **** 2019-12-11T16:26:01.174 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.218 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.083) 0:02:10.820 **** 2019-12-11T16:26:01.251 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.285 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.066) 0:02:10.886 **** 2019-12-11T16:26:01.311 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-11T16:26:01.313 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.351 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.065) 0:02:10.952 **** 2019-12-11T16:26:01.367 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* 2019-12-11T16:26:01.368 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-11T16:26:01.406 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.055) 0:02:11.007 **** 2019-12-11T16:26:01.412 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.452 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.046) 0:02:11.054 **** 2019-12-11T16:26:01.458 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-11T16:26:01.458 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.501 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.048) 0:02:11.103 **** 2019-12-11T16:26:01.507 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-11T16:26:01.509 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.547 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.045) 0:02:11.148 **** 2019-12-11T16:26:01.552 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-12-11T16:26:01.556 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.599 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.052) 0:02:11.200 **** 2019-12-11T16:26:01.607 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.646 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.046) 0:02:11.247 **** 2019-12-11T16:26:01.648 INFO:teuthology.task.ansible.out: 2019-12-11T16:26:01.652 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-11T16:26:01.653 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.703 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.049) 0:02:11.297 **** TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.748 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.052) 0:02:11.349 **** 2019-12-11T16:26:01.759 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.796 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.046) 0:02:11.396 **** 2019-12-11T16:26:01.802 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-11T16:26:01.804 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.843 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.047) 0:02:11.444 **** 2019-12-11T16:26:01.850 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-11T16:26:01.851 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.889 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.046) 0:02:11.490 **** 2019-12-11T16:26:01.915 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:01.968 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:01 +0000 (0:00:00.078) 0:02:11.569 **** 2019-12-11T16:26:02.008 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-12-11T16:26:02.009 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:02.076 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:02 +0000 (0:00:00.107) 0:02:11.677 **** 2019-12-11T16:26:02.116 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-11T16:26:02.117 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:02.174 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:02 +0000 (0:00:00.097) 0:02:11.775 **** 2019-12-11T16:26:03.530 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-12-11T16:26:03.532 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.353626", "end": "2019-12-11 16:26:03.500456", "rc": 0, "start": "2019-12-11 16:26:02.146830", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:26:03.534 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.034561", "end": "2019-12-11 16:26:03.500463", "rc": 0, "start": "2019-12-11 16:26:02.465902", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:26:03.572 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:03 +0000 (0:00:01.398) 0:02:13.173 **** 2019-12-11T16:26:03.607 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:03 +0000 (0:00:00.034) 0:02:13.208 **** 2019-12-11T16:26:04.406 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-12-11T16:26:04.408 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-12-11T16:26:04.449 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:04 +0000 (0:00:00.841) 0:02:14.050 **** 2019-12-11T16:26:04.476 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:04.524 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:04 +0000 (0:00:00.074) 0:02:14.125 **** 2019-12-11T16:26:04.550 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-12-11T16:26:04.551 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:04.590 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:04 +0000 (0:00:00.066) 0:02:14.191 **** 2019-12-11T16:26:04.620 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:04.657 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:04 +0000 (0:00:00.067) 0:02:14.258 **** 2019-12-11T16:26:04.683 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-11T16:26:04.685 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:04.723 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:04 +0000 (0:00:00.065) 0:02:14.324 **** 2019-12-11T16:26:05.021 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-12-11T16:26:05.064 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:05 +0000 (0:00:00.341) 0:02:14.665 **** 2019-12-11T16:26:05.092 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-12-11T16:26:05.094 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:05.138 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:05 +0000 (0:00:00.073) 0:02:14.739 **** 2019-12-11T16:26:05.174 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-12-11T16:26:05.175 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:05.217 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:05 +0000 (0:00:00.079) 0:02:14.818 **** 2019-12-11T16:26:05.244 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:05.288 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:05 +0000 (0:00:00.071) 0:02:14.890 **** 2019-12-11T16:26:05.327 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-11T16:26:05.328 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:05.375 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:05 +0000 (0:00:00.087) 0:02:14.977 **** 2019-12-11T16:26:05.638 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-12-11T16:26:05.639 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-12-11T16:26:05.676 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:05 +0000 (0:00:00.300) 0:02:15.278 **** 2019-12-11T16:26:06.311 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-12-11T16:26:06.360 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:06 +0000 (0:00:00.677) 0:02:15.955 **** 2019-12-11T16:26:09.903 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-12-11T16:26:09.941 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:09 +0000 (0:00:03.587) 0:02:19.543 **** 2019-12-11T16:26:09.943 INFO:teuthology.task.ansible.out: 2019-12-11T16:26:10.372 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-12-11T16:26:10.415 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:10 +0000 (0:00:00.473) 0:02:20.016 **** 2019-12-11T16:26:10.928 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi199.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-12-11T16:26:11.375 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-12-11T16:26:11.906 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-12-11T16:26:12.301 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-12-11T16:26:12.738 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-12-11T16:26:12.780 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:12 +0000 (0:00:02.364) 0:02:22.381 **** 2019-12-11T16:26:13.063 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-11T16:26:13.064 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-11T16:26:13.102 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:13 +0000 (0:00:00.322) 0:02:22.703 **** 2019-12-11T16:26:13.187 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-12-11T16:26:13.188 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-12-11T16:26:13.232 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:13 +0000 (0:00:00.129) 0:02:22.833 **** 2019-12-11T16:26:13.543 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-11T16:26:13.544 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-11T16:26:13.587 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:13 +0000 (0:00:00.355) 0:02:23.188 **** 2019-12-11T16:26:14.047 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-11T16:26:14.047 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-12-11T16:26:14.510 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-12-11T16:26:14.945 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-12-11T16:26:15.325 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-12-11T16:26:15.835 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-12-11T16:26:15.886 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:15 +0000 (0:00:02.298) 0:02:25.487 **** 2019-12-11T16:26:15.922 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-12-11T16:26:15.924 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:15.967 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:15 +0000 (0:00:00.081) 0:02:25.569 **** 2019-12-11T16:26:16.007 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-12-11T16:26:16.009 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-12-11T16:26:16.047 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.079) 0:02:25.648 **** 2019-12-11T16:26:16.211 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-11T16:26:16.212 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-12-11T16:26:16.261 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.213) 0:02:25.862 **** 2019-12-11T16:26:16.298 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-12-11T16:26:16.299 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.344 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.082) 0:02:25.945 **** 2019-12-11T16:26:16.360 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-11T16:26:16.362 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-11T16:26:16.405 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.058) 0:02:26.004 **** 2019-12-11T16:26:16.411 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-12-11T16:26:16.450 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.047) 0:02:26.052 **** 2019-12-11T16:26:16.456 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-12-11T16:26:16.460 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.461 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.473 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.496 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.523 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.072) 0:02:26.125 **** 2019-12-11T16:26:16.788 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-11T16:26:16.790 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-11T16:26:16.827 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.304) 0:02:26.429 **** 2019-12-11T16:26:16.878 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-12-11T16:26:16.879 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.890 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.903 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.912 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-11T16:26:16.952 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:16 +0000 (0:00:00.124) 0:02:26.554 **** 2019-12-11T16:26:19.907 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-12-11T16:26:19.907 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2019-12-11T16:26:19.946 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:19 +0000 (0:00:02.993) 0:02:29.547 **** 2019-12-11T16:26:19.972 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-12-11T16:26:19.973 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:20.009 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:20 +0000 (0:00:00.063) 0:02:29.611 **** 2019-12-11T16:26:20.343 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-12-11T16:26:20.345 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2019-12-11T16:26:20.393 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:20 +0000 (0:00:00.377) 0:02:29.988 **** 2019-12-11T16:26:20.426 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-12-11T16:26:20.428 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:20.483 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:20 +0000 (0:00:00.095) 0:02:30.084 **** 2019-12-11T16:26:20.938 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-12-11T16:26:20.940 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-12-11T16:26:20.981 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:20 +0000 (0:00:00.498) 0:02:30.583 **** 2019-12-11T16:26:21.294 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-12-11T16:26:21.377 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:21 +0000 (0:00:00.390) 0:02:30.973 **** 2019-12-11T16:26:21.942 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-12-11T16:26:22.011 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:22 +0000 (0:00:00.639) 0:02:31.613 **** 2019-12-11T16:26:22.039 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-12-11T16:26:22.041 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:22.080 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:22 +0000 (0:00:00.068) 0:02:31.681 **** 2019-12-11T16:26:22.278 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-12-11T16:26:22.319 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:22 +0000 (0:00:00.239) 0:02:31.920 **** 2019-12-11T16:26:22.355 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:22.403 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:22 +0000 (0:00:00.083) 0:02:32.004 **** 2019-12-11T16:26:22.430 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-12-11T16:26:22.432 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:26:22.580 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:22 +0000 (0:00:00.176) 0:02:32.180 **** 2019-12-11T16:26:22.848 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:26:22.883 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:22 +0000 (0:00:00.304) 0:02:32.484 **** 2019-12-11T16:26:22.934 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-12-11T16:26:22.935 INFO:teuthology.task.ansible.out:fatal: [smithi199.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-12-11T16:26:22.980 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:22 +0000 (0:00:00.096) 0:02:32.581 **** 2019-12-11T16:26:23.007 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-12-11T16:26:23.009 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:23.063 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:23 +0000 (0:00:00.082) 0:02:32.663 **** 2019-12-11T16:26:23.240 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-12-11T16:26:23.242 INFO:teuthology.task.ansible.out:fatal: [smithi079.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-12-11T16:26:23.284 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:23 +0000 (0:00:00.221) 0:02:32.885 **** 2019-12-11T16:26:23.320 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-12-11T16:26:23.322 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:26:23.378 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:23 +0000 (0:00:00.094) 0:02:32.979 **** 2019-12-11T16:26:24.035 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-12-11T16:26:24.037 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-12-11T16:26:24.082 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:24 +0000 (0:00:00.703) 0:02:33.683 **** 2019-12-11T16:26:24.275 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-12-11T16:26:24.328 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:24 +0000 (0:00:00.246) 0:02:33.930 **** 2019-12-11T16:26:24.935 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-12-11T16:26:24.936 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2019-12-11 16:23:29 UTC", "ActiveEnterTimestampMonotonic": "20630930", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target local-fs.target basic.target systemd-journald.socket var-run.mount time-sync.target system.slice nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2019-12-11 16:23:29 UTC", "AssertTimestampMonotonic": "20629784", "Before": "xdm.service shutdown.target getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2019-12-11 16:23:29 UTC", "ConditionTimestampMonotonic": "20629783", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8958", "ExecMainStartTimestamp": "Wed 2019-12-11 16:23:29 UTC", "ExecMainStartTimestampMonotonic": "20630878", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2019-12-11T16:26:24.938 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Wed 2019-12-11 16:23:29 UTC] ; stop_time=[n/a] ; pid=8958 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2019-12-11 16:23:29 UTC", "InactiveExitTimestampMonotonic": "20630930", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8958", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2019-12-11T16:26:24.939 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Wed 2019-12-11 16:23:29 UTC", "WatchdogTimestampMonotonic": "20630913", "WatchdogUSec": "0"}} 2019-12-11T16:26:24.977 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:24 +0000 (0:00:00.648) 0:02:34.578 **** 2019-12-11T16:26:24.986 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2019-12-11 16:23:16 UTC", "ActiveEnterTimestampMonotonic": "20965898", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount nss-lookup.target systemd-journald.socket basic.target network.target time-sync.target local-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2019-12-11 16:23:16 UTC", "AssertTimestampMonotonic": "20965194", "Before": "getty@tty1.service shutdown.target xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2019-12-11 16:23:16 UTC", "ConditionTimestampMonotonic": "20965193", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8932", "ExecMainStartTimestamp": "Wed 2019-12-11 16:23:16 UTC", "ExecMainStartTimestampMonotonic": "20965848", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2019-12-11T16:26:24.988 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Wed 2019-12-11 16:23:16 UTC] ; stop_time=[n/a] ; pid=8932 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2019-12-11 16:23:16 UTC", "InactiveExitTimestampMonotonic": "20965898", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8932", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2019-12-11T16:26:24.989 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Wed 2019-12-11 16:23:16 UTC", "WatchdogTimestampMonotonic": "20965881", "WatchdogUSec": "0"}} 2019-12-11T16:26:25.031 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:25 +0000 (0:00:00.053) 0:02:34.632 **** 2019-12-11T16:26:25.307 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.004368", "end": "2019-12-11 16:26:25.264405", "rc": 0, "start": "2019-12-11 16:26:25.260037", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-12-11T16:26:25.339 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:25 +0000 (0:00:00.306) 0:02:34.939 **** 2019-12-11T16:26:25.360 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.004864", "end": "2019-12-11 16:26:25.328916", "rc": 0, "start": "2019-12-11 16:26:25.324052", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-12-11T16:26:25.394 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:25 +0000 (0:00:00.056) 0:02:34.995 **** 2019-12-11T16:26:25.822 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-12-11T16:26:25.824 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-12-11T16:26:25.871 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:25 +0000 (0:00:00.476) 0:02:35.472 **** 2019-12-11T16:26:25.878 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-12-11T16:26:25.925 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:25 +0000 (0:00:00.054) 0:02:35.527 **** 2019-12-11T16:26:51.896 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-12-11T16:26:51.896 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7_7.6.noarch from updates"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7_7.6 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7_7.6 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 2019-12-11T16:26:51.898 INFO:teuthology.task.ansible.out: 24 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n Verifying : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7_7.6 \n\nComplete!\n"]} 2019-12-11T16:26:51.941 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:51 +0000 (0:00:26.015) 0:03:01.542 **** 2019-12-11T16:26:52.639 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-12-11T16:26:52.640 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-12-11T16:26:52.692 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:52 +0000 (0:00:00.751) 0:03:02.294 **** 2019-12-11T16:26:53.134 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-12-11T16:26:53.135 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7_7.6.noarch from updates"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7_7.6 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7_7.6 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 2019-12-11T16:26:53.137 INFO:teuthology.task.ansible.out: 23 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n Verifying : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7_7.6 \n\nComplete!\n"]} 2019-12-11T16:26:53.174 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:53 +0000 (0:00:00.480) 0:03:02.774 **** 2019-12-11T16:26:53.735 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-12-11T16:26:53.737 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-12-11T16:26:53.789 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:26:53 +0000 (0:00:00.614) 0:03:03.389 **** 2019-12-11T16:27:01.094 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:08.072898", "end": "2019-12-11 16:27:01.052280", "failed_when_result": false, "rc": 0, "start": "2019-12-11 16:26:52.979382", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-12-11T16:27:01.133 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:01 +0000 (0:00:07.345) 0:03:10.734 **** 2019-12-11T16:27:01.950 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:07.811676", "end": "2019-12-11 16:27:01.909073", "failed_when_result": false, "rc": 0, "start": "2019-12-11 16:26:54.097397", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-12-11T16:27:02.016 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:02 +0000 (0:00:00.881) 0:03:11.616 **** 2019-12-11T16:27:02.041 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-12-11T16:27:02.043 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-12-11T16:27:02.084 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:02 +0000 (0:00:00.069) 0:03:11.685 **** 2019-12-11T16:27:02.397 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-11T16:27:02.398 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013305", "end": "2019-12-11 16:27:02.375817", "rc": 0, "start": "2019-12-11 16:27:02.362512", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-12-11T16:27:02.446 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:02 +0000 (0:00:00.362) 0:03:12.048 **** 2019-12-11T16:27:02.777 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.017671", "end": "2019-12-11 16:27:02.751666", "rc": 0, "start": "2019-12-11 16:27:02.733995", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:27:02.823 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:02 +0000 (0:00:00.376) 0:03:12.424 **** 2019-12-11T16:27:02.937 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-12-11T16:27:02.939 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-12-11T16:27:03.017 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:03 +0000 (0:00:00.194) 0:03:12.618 **** 2019-12-11T16:27:03.386 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-11T16:27:03.387 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.020826", "end": "2019-12-11 16:27:03.355969", "rc": 0, "start": "2019-12-11 16:27:03.335143", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-12-11T16:27:03.426 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:03 +0000 (0:00:00.407) 0:03:13.026 **** 2019-12-11T16:27:03.780 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009992", "end": "2019-12-11 16:27:03.749407", "rc": 0, "start": "2019-12-11 16:27:03.739415", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:27:03.830 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:03 +0000 (0:00:00.405) 0:03:13.431 **** 2019-12-11T16:27:10.866 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-12-11T16:27:10.872 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:07.733414", "end": "2019-12-11 16:27:10.834700", "rc": 0, "start": "2019-12-11 16:27:03.101286", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:27:10.911 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:10 +0000 (0:00:07.080) 0:03:20.512 **** 2019-12-11T16:27:11.272 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-12-11T16:27:11.315 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:11 +0000 (0:00:00.399) 0:03:20.912 **** 2019-12-11T16:27:11.350 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-12-11T16:27:11.352 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-12-11T16:27:11.391 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:11 +0000 (0:00:00.076) 0:03:20.988 **** 2019-12-11T16:27:11.427 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi079.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-11T16:27:11.465 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:11 +0000 (0:00:00.078) 0:03:21.066 **** 2019-12-11T16:27:11.502 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-12-11T16:27:11.504 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-12-11T16:27:11.553 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:11 +0000 (0:00:00.087) 0:03:21.154 **** 2019-12-11T16:27:11.600 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-12-11T16:27:11.603 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-11T16:27:11.643 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:11 +0000 (0:00:00.088) 0:03:21.243 **** 2019-12-11T16:27:11.950 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-11T16:27:11.996 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:11 +0000 (0:00:00.352) 0:03:21.595 **** 2019-12-11T16:27:12.200 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-12-11T16:27:12.201 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:08.047249", "end": "2019-12-11 16:27:12.169591", "rc": 0, "start": "2019-12-11 16:27:04.122342", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:27:12.239 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.245) 0:03:21.841 **** 2019-12-11T16:27:12.332 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-12-11T16:27:12.377 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.138) 0:03:21.979 **** 2019-12-11T16:27:12.406 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-12-11T16:27:12.409 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:12.447 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.069) 0:03:22.049 **** 2019-12-11T16:27:12.474 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-12-11T16:27:12.475 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:12.513 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.065) 0:03:22.114 **** 2019-12-11T16:27:12.606 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-12-11T16:27:12.645 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.131) 0:03:22.246 **** 2019-12-11T16:27:12.694 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-12-11T16:27:12.695 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-12-11T16:27:12.748 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.101) 0:03:22.348 **** 2019-12-11T16:27:12.799 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-12-11T16:27:12.801 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-11T16:27:12.839 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.091) 0:03:22.440 **** 2019-12-11T16:27:12.876 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-12-11T16:27:12.877 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-12-11T16:27:12.881 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-12-11T16:27:12.882 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-12-11T16:27:12.932 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.093) 0:03:22.533 **** 2019-12-11T16:27:12.977 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:12 +0000 (0:00:00.045) 0:03:22.579 **** 2019-12-11T16:27:13.038 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-12-11T16:27:13.039 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-11T16:27:13.088 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:13 +0000 (0:00:00.110) 0:03:22.689 **** 2019-12-11T16:27:13.439 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-11T16:27:13.483 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:13 +0000 (0:00:00.395) 0:03:23.085 **** 2019-12-11T16:27:13.835 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-12-11T16:27:13.837 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-12-11T16:27:13.873 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:13 +0000 (0:00:00.389) 0:03:23.474 **** 2019-12-11T16:27:13.900 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:13.939 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:13 +0000 (0:00:00.065) 0:03:23.540 **** 2019-12-11T16:27:13.968 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-12-11T16:27:13.970 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:14.012 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:14 +0000 (0:00:00.069) 0:03:23.610 **** 2019-12-11T16:27:14.015 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-12-11T16:27:14.019 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-12-11T16:27:14.063 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:14 +0000 (0:00:00.054) 0:03:23.665 **** 2019-12-11T16:27:14.101 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-12-11T16:27:14.103 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi079"}, "changed": false} 2019-12-11T16:27:14.144 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:14 +0000 (0:00:00.076) 0:03:23.741 **** 2019-12-11T16:27:14.377 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi199.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-12-11T16:27:14.449 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:14 +0000 (0:00:00.309) 0:03:24.050 **** 2019-12-11T16:27:15.136 INFO:teuthology.task.ansible.out: 2019-12-11T16:27:15.139 INFO:teuthology.task.ansible.out:TASK [testnode : Set the system's hostname] ************************************ 2019-12-11T16:27:15.140 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi079.front.sepia.ceph.com", "ansible_hostname": "smithi079", "ansible_nodename": "smithi079"}, "changed": true, "name": "smithi079"} 2019-12-11T16:27:15.141 INFO:teuthology.task.ansible.out: 2019-12-11T16:27:15.195 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:15 +0000 (0:00:00.745) 0:03:24.796 **** 2019-12-11T16:27:15.510 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-12-11T16:27:15.511 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-12-11T16:27:15.565 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:15 +0000 (0:00:00.370) 0:03:25.167 **** 2019-12-11T16:27:15.603 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-12-11T16:27:15.605 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi199"}, "changed": false} 2019-12-11T16:27:15.646 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:15 +0000 (0:00:00.079) 0:03:25.247 **** 2019-12-11T16:27:16.086 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-12-11T16:27:16.140 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:16 +0000 (0:00:00.494) 0:03:25.741 **** 2019-12-11T16:27:16.204 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi199.front.sepia.ceph.com", "ansible_hostname": "smithi199", "ansible_nodename": "smithi199"}, "changed": true, "name": "smithi199"} 2019-12-11T16:27:16.242 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:16 +0000 (0:00:00.102) 0:03:25.844 **** 2019-12-11T16:27:17.022 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-12-11T16:27:17.092 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:17 +0000 (0:00:00.849) 0:03:26.694 **** 2019-12-11T16:27:17.100 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-12-11T16:27:17.102 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-12-11T16:27:17.158 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:17 +0000 (0:00:00.065) 0:03:26.759 **** 2019-12-11T16:27:18.046 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-12-11T16:27:18.084 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:18 +0000 (0:00:00.926) 0:03:27.686 **** 2019-12-11T16:27:18.091 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-12-11T16:27:18.092 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-11T16:27:18.141 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:18 +0000 (0:00:00.055) 0:03:27.742 **** 2019-12-11T16:27:18.977 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-11T16:27:19.037 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:19 +0000 (0:00:00.896) 0:03:28.638 **** 2019-12-11T16:27:19.044 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-12-11T16:27:19.088 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:19 +0000 (0:00:00.046) 0:03:28.685 **** 2019-12-11T16:27:19.405 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-11T16:27:19.406 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-11T16:27:19.460 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:19 +0000 (0:00:00.375) 0:03:29.060 **** 2019-12-11T16:27:19.763 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-11T16:27:19.764 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 10103, "name": "kvm", "state": "present", "system": false} 2019-12-11T16:27:19.805 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:19 +0000 (0:00:00.343) 0:03:29.404 **** 2019-12-11T16:27:19.967 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-12-11T16:27:19.969 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-12-11T16:27:20.013 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:20 +0000 (0:00:00.203) 0:03:29.607 **** 2019-12-11T16:27:20.162 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-11T16:27:20.163 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-11T16:27:20.215 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:20 +0000 (0:00:00.209) 0:03:29.816 **** 2019-12-11T16:27:20.328 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-11T16:27:20.330 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-11T16:27:20.375 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:20 +0000 (0:00:00.159) 0:03:29.976 **** 2019-12-11T16:27:20.684 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-11T16:27:20.686 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 10103, "name": "kvm", "state": "present", "system": false} 2019-12-11T16:27:20.731 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:20 +0000 (0:00:00.356) 0:03:30.332 **** 2019-12-11T16:27:21.096 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi199.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-11T16:27:21.137 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:21 +0000 (0:00:00.406) 0:03:30.739 **** 2019-12-11T16:27:21.166 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-11T16:27:21.167 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-12-11T16:27:21.215 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:21 +0000 (0:00:00.077) 0:03:30.816 **** 2019-12-11T16:27:22.064 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-12-11T16:27:22.103 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:22 +0000 (0:00:00.883) 0:03:31.700 **** 2019-12-11T16:27:22.169 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-12-11T16:27:22.172 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-12-11T16:27:22.214 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:22 +0000 (0:00:00.115) 0:03:31.816 **** 2019-12-11T16:27:23.035 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-12-11T16:27:23.084 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:23 +0000 (0:00:00.863) 0:03:32.679 **** 2019-12-11T16:27:24.700 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-11T16:27:24.701 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-12-11T16:27:24.703 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.171893", "end": "2019-12-11 16:27:24.660785", "rc": 0, "start": "2019-12-11 16:27:22.488892", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:27:24.753 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:24 +0000 (0:00:01.674) 0:03:34.354 **** 2019-12-11T16:27:24.781 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-12-11T16:27:24.821 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:24 +0000 (0:00:00.067) 0:03:34.422 **** 2019-12-11T16:27:25.569 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-11T16:27:25.570 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.156133", "end": "2019-12-11 16:27:25.539567", "rc": 0, "start": "2019-12-11 16:27:23.383434", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:27:25.609 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:25 +0000 (0:00:00.788) 0:03:35.210 **** 2019-12-11T16:27:25.636 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-12-11T16:27:25.686 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:25 +0000 (0:00:00.077) 0:03:35.288 **** 2019-12-11T16:27:25.868 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi079.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2019-12-11T16:27:26.699 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2019-12-11T16:27:26.851 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2019-12-11T16:27:26.917 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:26 +0000 (0:00:01.230) 0:03:36.518 **** 2019-12-11T16:27:26.954 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-12-11T16:27:26.956 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:26.997 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:26 +0000 (0:00:00.080) 0:03:36.598 **** 2019-12-11T16:27:27.037 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-12-11T16:27:27.040 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-11T16:27:27.051 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-11T16:27:27.113 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.115) 0:03:36.714 **** 2019-12-11T16:27:27.152 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-11T16:27:27.153 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:27.191 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.078) 0:03:36.793 **** 2019-12-11T16:27:27.231 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-12-11T16:27:27.270 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.075) 0:03:36.868 **** 2019-12-11T16:27:27.628 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi199.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2019-12-11T16:27:27.688 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.420) 0:03:37.288 **** 2019-12-11T16:27:27.715 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:27.779 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.092) 0:03:37.381 **** 2019-12-11T16:27:27.797 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-11T16:27:27.798 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-12-11T16:27:27.841 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.062) 0:03:37.443 **** 2019-12-11T16:27:27.850 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-12-11T16:27:27.851 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-11T16:27:27.853 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-11T16:27:27.897 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.055) 0:03:37.498 **** 2019-12-11T16:27:27.935 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-11T16:27:27.937 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:27.977 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:27 +0000 (0:00:00.080) 0:03:37.578 **** 2019-12-11T16:27:28.014 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-11T16:27:28.016 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-12-11T16:27:28.057 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:28 +0000 (0:00:00.075) 0:03:37.654 **** 2019-12-11T16:27:28.499 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-12-11T16:27:28.501 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.327415", "end": "2019-12-11 16:27:28.465589", "rc": 0, "start": "2019-12-11 16:27:28.138174", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.facebook.net", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: mirror.facebook.net", " * updates: centos.mirror.constant.com", "No unfinished transactions left."]} 2019-12-11T16:27:28.541 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:28 +0000 (0:00:00.486) 0:03:38.141 **** 2019-12-11T16:27:28.549 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-11T16:27:28.549 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-12-11T16:27:28.599 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:28 +0000 (0:00:00.058) 0:03:38.200 **** 2019-12-11T16:27:28.942 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-12-11T16:27:28.943 INFO:teuthology.task.ansible.out:fatal: [smithi079.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.045415", "end": "2019-12-11 16:27:28.912212", "msg": "non-zero return code", "rc": 1, "start": "2019-12-11 16:27:28.866797", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-12-11T16:27:28.981 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:28 +0000 (0:00:00.382) 0:03:38.582 **** 2019-12-11T16:27:29.019 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:29.069 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:29 +0000 (0:00:00.086) 0:03:38.669 **** 2019-12-11T16:27:29.283 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.342913", "end": "2019-12-11 16:27:29.237613", "rc": 0, "start": "2019-12-11 16:27:28.894700", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.facebook.net", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: mirror.facebook.net", " * updates: centos.mirror.constant.com", "No unfinished transactions left."]} 2019-12-11T16:27:29.340 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:29 +0000 (0:00:00.267) 0:03:38.936 **** 2019-12-11T16:27:29.729 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi199.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.056607", "end": "2019-12-11 16:27:29.698429", "msg": "non-zero return code", "rc": 1, "start": "2019-12-11 16:27:29.641822", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-12-11T16:27:29.792 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:29 +0000 (0:00:00.455) 0:03:39.391 **** 2019-12-11T16:27:29.819 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-12-11T16:27:29.820 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:27:29.870 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:29 +0000 (0:00:00.080) 0:03:39.472 **** 2019-12-11T16:27:37.510 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 2019-12-11T16:27:37.512 INFO:teuthology.task.ansible.out: @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemov 2019-12-11T16:27:37.513 INFO:teuthology.task.ansible.out:ed:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-11T16:27:37.560 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:37 +0000 (0:00:07.689) 0:03:47.161 **** 2019-12-11T16:27:38.139 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 2019-12-11T16:27:38.140 INFO:teuthology.task.ansible.out: @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos 2019-12-11T16:27:38.145 INFO:teuthology.task.ansible.out: \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-11T16:27:38.182 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:38 +0000 (0:00:00.619) 0:03:47.781 **** 2019-12-11T16:27:41.148 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-11T16:27:41.149 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-12-11T16:27:41.199 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:41 +0000 (0:00:03.019) 0:03:50.800 **** 2019-12-11T16:27:41.525 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-12-11T16:27:41.570 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:41 +0000 (0:00:00.370) 0:03:51.171 **** 2019-12-11T16:27:59.637 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-11T16:27:59.638 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": ["@core", "@base", "iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "Warning: Group core does not have any packages to install.\nWarning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-20.el7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is alrea 2019-12-11T16:27:59.640 INFO:teuthology.task.ansible.out:dy installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already insta 2019-12-11T16:27:59.648 INFO:teuthology.task.ansible.out:lled", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies R 2019-12-11T16:27:59.652 INFO:teuthology.task.ansible.out:esolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================================================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 25 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/8 \n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 2019-12-11T16:27:59.654 INFO:teuthology.task.ansible.out: 7/8 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el 2019-12-11T16:27:59.656 INFO:teuthology.task.ansible.out:7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-11T16:27:59.697 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:27:59 +0000 (0:00:18.126) 0:04:09.298 **** 2019-12-11T16:28:00.209 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": ["@core", "@base", "iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "Warning: Group core does not have any packages to install.\nWarning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-20.el7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is alrea 2019-12-11T16:28:00.211 INFO:teuthology.task.ansible.out:dy installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already insta 2019-12-11T16:28:00.212 INFO:teuthology.task.ansible.out:lled", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.facebook.net\n * updates: centos.mirror.constant.com\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies R 2019-12-11T16:28:00.217 INFO:teuthology.task.ansible.out:esolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================================================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 20 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/8 \n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 2019-12-11T16:28:00.219 INFO:teuthology.task.ansible.out: 7/8 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-11T16:28:00.251 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:00 +0000 (0:00:00.554) 0:04:09.853 **** 2019-12-11T16:28:00.440 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-12-11T16:28:00.441 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-12-11T16:28:00.485 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:00 +0000 (0:00:00.233) 0:04:10.087 **** 2019-12-11T16:28:00.978 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2019-12-11T16:28:01.032 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:01 +0000 (0:00:00.539) 0:04:10.627 **** TASK [testnode : Install epel packages] **************************************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-12-11T16:28:01.076 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:01 +0000 (0:00:00.048) 0:04:10.675 **** 2019-12-11T16:28:01.078 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-11T16:28:01.079 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:01.118 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:01 +0000 (0:00:00.044) 0:04:10.720 **** 2019-12-11T16:28:01.565 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-12-11T16:28:01.567 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service system.slice basic.target polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ebtables.service ipset.service shutdown.target ip6tables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2019-12-11T16:28:01.568 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2019-12-11T16:28:01.570 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-11T16:28:01.607 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:01 +0000 (0:00:00.488) 0:04:11.208 **** 2019-12-11T16:28:01.607 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:01.614 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2019-12-11T16:28:01.670 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:01 +0000 (0:00:00.063) 0:04:11.272 **** 2019-12-11T16:28:01.717 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-11T16:28:01.719 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:01.758 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:01 +0000 (0:00:00.088) 0:04:11.360 **** 2019-12-11T16:28:02.004 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-12-11T16:28:02.005 INFO:teuthology.task.ansible.out:fatal: [smithi079.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-12-11T16:28:02.041 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:02 +0000 (0:00:00.282) 0:04:11.643 **** 2019-12-11T16:28:02.203 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-12-11T16:28:02.204 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice polkit.service dbus.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ipset.service shutdown.target ebtables.service ip6tables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2019-12-11T16:28:02.206 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2019-12-11T16:28:02.208 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-11T16:28:02.252 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:02 +0000 (0:00:00.210) 0:04:11.853 **** 2019-12-11T16:28:02.680 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-12-11T16:28:02.682 INFO:teuthology.task.ansible.out:fatal: [smithi199.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-12-11T16:28:02.718 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:02 +0000 (0:00:00.466) 0:04:12.320 **** 2019-12-11T16:28:02.901 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-12-11T16:28:02.903 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-12-11T16:28:02.949 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:02 +0000 (0:00:00.219) 0:04:12.540 **** 2019-12-11T16:28:02.966 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-12-11T16:28:02.968 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.003 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:02 +0000 (0:00:00.064) 0:04:12.604 **** 2019-12-11T16:28:03.030 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-11T16:28:03.070 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.067) 0:04:12.672 **** 2019-12-11T16:28:03.097 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-12-11T16:28:03.099 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.137 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.066) 0:04:12.738 **** 2019-12-11T16:28:03.143 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-12-11T16:28:03.145 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-12-11T16:28:03.184 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.044) 0:04:12.783 **** 2019-12-11T16:28:03.187 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-11T16:28:03.189 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.226 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.044) 0:04:12.827 **** 2019-12-11T16:28:03.232 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-12-11T16:28:03.232 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.272 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.045) 0:04:12.873 **** 2019-12-11T16:28:03.276 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-12-11T16:28:03.277 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.278 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.334 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.062) 0:04:12.936 **** 2019-12-11T16:28:03.341 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-11T16:28:03.389 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.054) 0:04:12.990 **** 2019-12-11T16:28:03.394 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-12-11T16:28:03.395 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.436 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.047) 0:04:13.038 **** 2019-12-11T16:28:03.444 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.482 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.046) 0:04:13.084 **** 2019-12-11T16:28:03.488 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-11T16:28:03.534 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.051) 0:04:13.135 **** 2019-12-11T16:28:03.543 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-11T16:28:03.544 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.580 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.046) 0:04:13.181 **** 2019-12-11T16:28:03.586 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-12-11T16:28:03.586 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.625 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.044) 0:04:13.226 **** 2019-12-11T16:28:03.631 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi199.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.669 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.044) 0:04:13.271 **** 2019-12-11T16:28:03.676 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-12-11T16:28:03.677 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.715 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.045) 0:04:13.316 **** 2019-12-11T16:28:03.728 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.759 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.043) 0:04:13.360 **** 2019-12-11T16:28:03.767 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-12-11T16:28:03.769 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.811 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.047) 0:04:13.407 **** 2019-12-11T16:28:03.814 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-11T16:28:03.855 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.046) 0:04:13.454 **** 2019-12-11T16:28:03.858 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-11T16:28:03.860 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.903 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.050) 0:04:13.504 **** 2019-12-11T16:28:03.911 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.949 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.045) 0:04:13.550 **** 2019-12-11T16:28:03.954 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-12-11T16:28:03.955 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:03.992 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:03 +0000 (0:00:00.042) 0:04:13.593 **** 2019-12-11T16:28:03.999 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-12-11T16:28:04.001 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.036 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.044) 0:04:13.637 **** 2019-12-11T16:28:04.042 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-11T16:28:04.042 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.080 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.043) 0:04:13.681 **** 2019-12-11T16:28:04.085 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-12-11T16:28:04.087 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.134 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.054) 0:04:13.735 **** 2019-12-11T16:28:04.143 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.179 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.044) 0:04:13.780 **** 2019-12-11T16:28:04.185 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-11T16:28:04.186 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.244 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.050) 0:04:13.831 **** TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.276 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.046) 0:04:13.878 **** 2019-12-11T16:28:04.284 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.321 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.045) 0:04:13.923 **** 2019-12-11T16:28:04.329 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-11T16:28:04.330 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.387 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.065) 0:04:13.988 **** 2019-12-11T16:28:04.400 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.432 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.045) 0:04:14.033 **** 2019-12-11T16:28:04.438 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-11T16:28:04.438 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.476 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.044) 0:04:14.078 **** 2019-12-11T16:28:04.488 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.524 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.047) 0:04:14.125 **** 2019-12-11T16:28:04.531 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-12-11T16:28:04.533 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.571 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.047) 0:04:14.173 **** 2019-12-11T16:28:04.577 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-12-11T16:28:04.578 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.624 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.052) 0:04:14.225 **** 2019-12-11T16:28:04.633 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-12-11T16:28:04.637 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.674 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.049) 0:04:14.275 **** 2019-12-11T16:28:04.679 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-11T16:28:04.681 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.718 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.043) 0:04:14.319 **** 2019-12-11T16:28:04.735 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.765 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.047) 0:04:14.366 **** 2019-12-11T16:28:04.772 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-11T16:28:04.773 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.813 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.048) 0:04:14.415 **** 2019-12-11T16:28:04.820 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.862 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.048) 0:04:14.464 **** 2019-12-11T16:28:04.872 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.909 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.047) 0:04:14.511 **** 2019-12-11T16:28:04.915 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:04.955 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.045) 0:04:14.557 **** 2019-12-11T16:28:04.964 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-12-11T16:28:04.965 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.004 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:04 +0000 (0:00:00.046) 0:04:14.604 **** 2019-12-11T16:28:05.008 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-11T16:28:05.009 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.059 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.048) 0:04:14.652 **** TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.095 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.044) 0:04:14.697 **** 2019-12-11T16:28:05.104 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.141 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.045) 0:04:14.742 **** 2019-12-11T16:28:05.149 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-12-11T16:28:05.151 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.195 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.054) 0:04:14.797 **** 2019-12-11T16:28:05.203 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-11T16:28:05.205 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.242 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.046) 0:04:14.843 **** 2019-12-11T16:28:05.249 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-11T16:28:05.251 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.291 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.049) 0:04:14.893 **** 2019-12-11T16:28:05.296 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-12-11T16:28:05.297 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.337 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.046) 0:04:14.939 **** 2019-12-11T16:28:05.343 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-11T16:28:05.345 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.382 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.044) 0:04:14.983 **** 2019-12-11T16:28:05.386 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-11T16:28:05.389 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.425 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.042) 0:04:15.026 **** 2019-12-11T16:28:05.430 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-11T16:28:05.433 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.470 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.046) 0:04:15.072 **** 2019-12-11T16:28:05.476 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-11T16:28:05.476 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.515 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.044) 0:04:15.117 **** 2019-12-11T16:28:05.522 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.571 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.056) 0:04:15.173 **** 2019-12-11T16:28:05.579 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.623 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.051) 0:04:15.225 **** 2019-12-11T16:28:05.629 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-12-11T16:28:05.630 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.669 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.045) 0:04:15.270 **** 2019-12-11T16:28:05.680 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.724 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.055) 0:04:15.326 **** 2019-12-11T16:28:05.731 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-11T16:28:05.732 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.777 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.049) 0:04:15.375 **** 2019-12-11T16:28:05.780 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.822 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.044) 0:04:15.420 **** 2019-12-11T16:28:05.825 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-11T16:28:05.826 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.867 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.046) 0:04:15.466 **** 2019-12-11T16:28:05.876 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.910 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.044) 0:04:15.511 **** 2019-12-11T16:28:05.918 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-11T16:28:05.919 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.955 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.045) 0:04:15.556 **** 2019-12-11T16:28:05.960 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-11T16:28:05.961 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:05.999 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:05 +0000 (0:00:00.044) 0:04:15.600 **** 2019-12-11T16:28:06.005 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-11T16:28:06.006 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:06.044 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:06 +0000 (0:00:00.045) 0:04:15.646 **** 2019-12-11T16:28:06.073 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-11T16:28:06.075 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:06.122 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:06 +0000 (0:00:00.077) 0:04:15.724 **** 2019-12-11T16:28:06.150 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-11T16:28:06.152 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:06.194 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:06 +0000 (0:00:00.071) 0:04:15.795 **** 2019-12-11T16:28:06.231 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-11T16:28:06.233 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:06.277 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:06 +0000 (0:00:00.083) 0:04:15.879 **** 2019-12-11T16:28:06.333 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-12-11T16:28:06.372 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:06 +0000 (0:00:00.094) 0:04:15.973 **** 2019-12-11T16:28:06.613 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-12-11T16:28:06.658 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:06 +0000 (0:00:00.286) 0:04:16.259 **** 2019-12-11T16:28:06.709 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-11T16:28:06.749 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:06 +0000 (0:00:00.090) 0:04:16.350 **** 2019-12-11T16:28:07.003 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-11T16:28:07.057 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.308) 0:04:16.659 **** 2019-12-11T16:28:07.085 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. 2019-12-11T16:28:07.088 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-12-11T16:28:07.125 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.067) 0:04:16.726 **** 2019-12-11T16:28:07.156 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.196 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.071) 0:04:16.798 **** 2019-12-11T16:28:07.234 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.272 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.074) 0:04:16.873 **** 2019-12-11T16:28:07.302 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-11T16:28:07.303 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.356 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.084) 0:04:16.957 **** 2019-12-11T16:28:07.384 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-11T16:28:07.385 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-12-11T16:28:07.434 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.078) 0:04:17.036 **** 2019-12-11T16:28:07.441 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-11T16:28:07.443 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.494 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.059) 0:04:17.096 **** 2019-12-11T16:28:07.502 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-11T16:28:07.503 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.552 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.057) 0:04:17.153 **** 2019-12-11T16:28:07.560 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-11T16:28:07.562 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.606 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.054) 0:04:17.207 **** 2019-12-11T16:28:07.614 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-11T16:28:07.615 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.659 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.053) 0:04:17.260 **** 2019-12-11T16:28:07.664 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-11T16:28:07.665 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.705 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.046) 0:04:17.307 **** 2019-12-11T16:28:07.712 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-11T16:28:07.714 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.756 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.050) 0:04:17.357 **** 2019-12-11T16:28:07.764 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-11T16:28:07.765 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.803 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.046) 0:04:17.404 **** 2019-12-11T16:28:07.819 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.857 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.054) 0:04:17.459 **** 2019-12-11T16:28:07.864 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.904 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.046) 0:04:17.505 **** 2019-12-11T16:28:07.914 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:07.962 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:07 +0000 (0:00:00.057) 0:04:17.563 **** 2019-12-11T16:28:07.970 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-11T16:28:07.971 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.010 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.047) 0:04:17.611 **** 2019-12-11T16:28:08.019 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.057 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.046) 0:04:17.658 **** 2019-12-11T16:28:08.063 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-12-11T16:28:08.065 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.106 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.048) 0:04:17.707 **** 2019-12-11T16:28:08.114 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.158 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.049) 0:04:17.757 **** 2019-12-11T16:28:08.161 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-12-11T16:28:08.163 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.202 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.046) 0:04:17.804 **** 2019-12-11T16:28:08.209 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.250 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.047) 0:04:17.851 **** 2019-12-11T16:28:08.255 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-11T16:28:08.255 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.296 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.045) 0:04:17.897 **** 2019-12-11T16:28:08.302 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-11T16:28:08.304 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.358 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.061) 0:04:17.958 **** 2019-12-11T16:28:08.364 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-11T16:28:08.371 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.410 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.053) 0:04:18.012 **** 2019-12-11T16:28:08.417 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-12-11T16:28:08.419 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.457 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.046) 0:04:18.058 **** 2019-12-11T16:28:08.462 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-12-11T16:28:08.462 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.501 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.044) 0:04:18.103 **** 2019-12-11T16:28:08.507 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-12-11T16:28:08.509 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.557 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.055) 0:04:18.158 **** 2019-12-11T16:28:08.562 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-12-11T16:28:08.563 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.604 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.045) 0:04:18.204 **** 2019-12-11T16:28:08.613 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.649 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.046) 0:04:18.250 **** 2019-12-11T16:28:08.654 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-12-11T16:28:08.655 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.694 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.044) 0:04:18.295 **** 2019-12-11T16:28:08.700 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-11T16:28:08.703 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.740 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.046) 0:04:18.342 **** 2019-12-11T16:28:08.745 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-11T16:28:08.746 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.786 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.045) 0:04:18.387 **** 2019-12-11T16:28:08.793 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-12-11T16:28:08.795 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.839 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.050) 0:04:18.438 **** 2019-12-11T16:28:08.844 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.884 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.047) 0:04:18.485 **** 2019-12-11T16:28:08.890 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-12-11T16:28:08.891 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.931 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.046) 0:04:18.532 **** 2019-12-11T16:28:08.936 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-12-11T16:28:08.938 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:08.975 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:08 +0000 (0:00:00.044) 0:04:18.576 **** 2019-12-11T16:28:08.983 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.023 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.047) 0:04:18.624 **** 2019-12-11T16:28:09.028 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-12-11T16:28:09.029 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.080 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.057) 0:04:18.682 **** 2019-12-11T16:28:09.086 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-11T16:28:09.088 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.126 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.046) 0:04:18.728 **** 2019-12-11T16:28:09.132 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:09.133 INFO:teuthology.task.ansible.out:TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.134 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:09.181 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.054) 0:04:18.783 **** 2019-12-11T16:28:09.189 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-12-11T16:28:09.193 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.239 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.057) 0:04:18.840 **** 2019-12-11T16:28:09.245 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-12-11T16:28:09.245 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.296 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.048) 0:04:18.889 **** TASK [testnode : Stop collectl] ************************************************ skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.337 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.049) 0:04:18.939 **** 2019-12-11T16:28:09.343 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-11T16:28:09.344 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-11T16:28:09.383 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.045) 0:04:18.984 **** 2019-12-11T16:28:09.389 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-12-11T16:28:09.391 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.444 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.060) 0:04:19.045 **** 2019-12-11T16:28:09.460 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi079.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi079.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi079.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.466 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.477 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.493 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi079.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.529 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.084) 0:04:19.130 **** 2019-12-11T16:28:09.535 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.582 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.053) 0:04:19.183 **** 2019-12-11T16:28:09.589 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.628 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.046) 0:04:19.230 **** 2019-12-11T16:28:09.677 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-11T16:28:09.730 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.101) 0:04:19.332 **** 2019-12-11T16:28:09.739 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi079.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003591", "end": "2019-12-11 16:28:09.707042", "rc": 0, "start": "2019-12-11 16:28:09.703451", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-11T16:28:09.785 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.054) 0:04:19.386 **** 2019-12-11T16:28:09.791 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-11T16:28:09.793 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.806 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi199.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.825 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-11T16:28:09.828 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi199.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.879 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.093) 0:04:19.480 **** 2019-12-11T16:28:09.884 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.885 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.887 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.933 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.054) 0:04:19.534 **** 2019-12-11T16:28:09.938 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-11T16:28:09.939 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:09.981 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:09 +0000 (0:00:00.048) 0:04:19.583 **** 2019-12-11T16:28:10.009 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:10.048 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.066) 0:04:19.649 **** 2019-12-11T16:28:10.075 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-12-11T16:28:10.076 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:10.116 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.068) 0:04:19.717 **** 2019-12-11T16:28:10.123 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-12-11T16:28:10.125 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003941", "end": "2019-12-11 16:28:10.056179", "rc": 0, "start": "2019-12-11 16:28:10.052238", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-11T16:28:10.168 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.051) 0:04:19.769 **** 2019-12-11T16:28:10.205 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-11T16:28:10.207 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:10.249 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.081) 0:04:19.850 **** 2019-12-11T16:28:10.286 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-11T16:28:10.288 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:10.326 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.077) 0:04:19.928 **** 2019-12-11T16:28:10.358 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:10.393 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.067) 0:04:19.995 **** 2019-12-11T16:28:10.421 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-12-11T16:28:10.422 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:10.459 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.065) 0:04:20.061 **** 2019-12-11T16:28:10.595 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-12-11T16:28:10.597 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-12-11T16:28:10.635 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.175) 0:04:20.237 **** 2019-12-11T16:28:10.680 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-11T16:28:10.682 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item={u'block_used': 13037528, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227286956, u'size_available': 930967371776, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966955, u'device': u'/dev/sda1', u'inode_used': 88021, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227286956, "block_size": 4096, "block_total": 240324484, "block_used": 13037528, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966955, "inode_total": 61054976, "inode_used": 88021, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930967371776, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}} 2019-12-11T16:28:10.721 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:10 +0000 (0:00:00.085) 0:04:20.322 **** 2019-12-11T16:28:10.969 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-12-11T16:28:10.969 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-12-11T16:28:11.026 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.304) 0:04:20.627 **** 2019-12-11T16:28:11.064 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-12-11T16:28:11.069 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.011611", "end": "2019-12-11 16:28:11.032325", "rc": 0, "start": "2019-12-11 16:28:11.020714", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:11.120 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.093) 0:04:20.721 **** 2019-12-11T16:28:11.130 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-11T16:28:11.132 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item={u'block_used': 13037525, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227286959, u'size_available': 930967384064, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966955, u'device': u'/dev/sda1', u'inode_used': 88021, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227286959, "block_size": 4096, "block_total": 240324484, "block_used": 13037525, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966955, "inode_total": 61054976, "inode_used": 88021, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930967384064, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}} 2019-12-11T16:28:11.179 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.059) 0:04:20.781 **** 2019-12-11T16:28:11.185 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-11T16:28:11.186 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'block_used': 13037528, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227286956, u'size_available': 930967371776, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966955, u'device': u'/dev/sda1', u'inode_used': 88021, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227286956, "block_size": 4096, "block_total": 240324484, "block_used": 13037528, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966955, "inode_total": 61054976, "inode_used": 88021, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930967371776, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:11.225 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.046) 0:04:20.827 **** 2019-12-11T16:28:11.260 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-11T16:28:11.295 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.069) 0:04:20.896 **** 2019-12-11T16:28:11.359 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-11T16:28:11.364 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QDP6', u'wwn-0x5000c50091e0487b'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QDP6', u'wwn': u'0x5000c50091e0487b', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QDP6-part1', u'wwn-0x5000c50091e0487b-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QDP6", "wwn-0x5000c50091e0487b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QDP6-part1", "wwn-0x5000c50091e0487b-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QDP6", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-11T16:28:11.366 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50091e0487b"}}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:11.470 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-12-11T16:28:11.472 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010963", "end": "2019-12-11 16:28:11.437731", "rc": 0, "start": "2019-12-11 16:28:11.426768", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:11.522 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.226) 0:04:21.123 **** 2019-12-11T16:28:11.559 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-11T16:28:11.560 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'block_used': 13037525, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227286959, u'size_available': 930967384064, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966955, u'device': u'/dev/sda1', u'inode_used': 88021, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227286959, "block_size": 4096, "block_total": 240324484, "block_used": 13037525, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966955, "inode_total": 61054976, "inode_used": 88021, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930967384064, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:11.629 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.106) 0:04:21.230 **** 2019-12-11T16:28:11.657 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-11T16:28:11.699 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:11 +0000 (0:00:00.066) 0:04:21.297 **** 2019-12-11T16:28:11.744 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-11T16:28:11.745 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XPB7', u'wwn-0x5000c5009292944a'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XPB7', u'wwn': u'0x5000c5009292944a', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XPB7-part1', u'wwn-0x5000c5009292944a-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPB7", "wwn-0x5000c5009292944a"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPB7-part1", "wwn-0x5000c5009292944a-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XPB7", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-11T16:28:11.747 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009292944a"}}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:13.252 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-lhGyfM-85Tr-VXc9-nxok-ppxd-sfxn-n4qrPo', u'nvme-INTEL_SSDPEDMD400G4_PHFT6204013G400BGN', u'nvme-nvme.8086-50484654363230343031334734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.577329", "end": "2019-12-11 16:28:13.209049", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-lhGyfM-85Tr-VXc9-nxok-ppxd-sfxn-n4qrPo", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013G400BGN", "nvme-nvme.8086-50484654363230343031334734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:11.631 2019-12-11T16:28:13.260 INFO:teuthology.task.ansible.out:720", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:13.509 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-IysFJx-yV3m-EZ1g-Z2pD-rLXa-0KE9-ri31PU', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300AG400BGN', u'nvme-nvme.8086-43564654363233333030414734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.461482", "end": "2019-12-11 16:28:13.469304", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-IysFJx-yV3m-EZ1g-Z2pD-rLXa-0KE9-ri31PU", "nvme-INTEL_SSDPEDMD400G4_CVFT623300AG400BGN", "nvme-nvme.8086-43564654363233333030414734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:12.007 2019-12-11T16:28:13.510 INFO:teuthology.task.ansible.out:822", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:14.627 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7a0Uy2NHomSPuLZsOXgBeZmfV3UWF0N35'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.059842", "end": "2019-12-11 16:28:14.588171", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7a0Uy2NHomSPuLZsOXgBeZmfV3UWF0N35"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:13.528329", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:14.872 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqAR8rWIInwBkAd3DG1x8aHs50DEtcyRAK'], u'uuids': [u'4bf739d0-db1a-46b8-8d24-c01986f1a2a8']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052508", "end": "2019-12-11 16:28:14.834220", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqAR8rWIInwBkAd3DG1x8aHs50DEtcyRAK"], "labels": [], "masters": [], "uuids": ["4bf739d0-db1a-46b8-8d24-c01986f1a2a8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:13.781712", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:15.988 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7iXZAxtLjtvvIPNN0znuBL3N0VZcYLyQK'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.067685", "end": "2019-12-11 16:28:15.955195", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7iXZAxtLjtvvIPNN0znuBL3N0VZcYLyQK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:14.887510", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:16.237 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqosTSy2AJ8JJVagTkFznEx4uN9m3gOywS'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.067161", "end": "2019-12-11 16:28:16.202374", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqosTSy2AJ8JJVagTkFznEx4uN9m3gOywS"], "labels": [], "masters": [], "uuids": []}, "model": null, "par 2019-12-11T16:28:16.237 INFO:teuthology.task.ansible.out:titions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:15.135213", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:17.381 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk72KjxnfuUApEiqlg53cXqJhU2gSeEHjkB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.074958", "end": "2019-12-11 16:28:17.347039", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk72KjxnfuUApEiqlg53cXqJhU2gSeEHjkB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:16.272081", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:17.633 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqHInHXf61aC32VFcgf0khClYsP6dXYmAQ'], u'uuids': [u'32186ca3-28d8-417e-9dfb-fe44207f5704']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.105534", "end": "2019-12-11 16:28:17.600513", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqHInHXf61aC32VFcgf0khClYsP6dXYmAQ"], "labels": [], "masters": [], "uuids": ["32186ca3-28d8-417e-9dfb-fe44207f5704"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:16.494979", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:18.748 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7cKC1QCl0IbuFo9UbFn42Q2AZj9kKEwyj'], u'uuids': [u'41afee2a-9e5e-4481-9743-c36a0320e2d5']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.059616", "end": "2019-12-11 16:28:18.713831", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7cKC1QCl0IbuFo9UbFn42Q2AZj9kKEwyj"], "labels": [], "masters": [], "uuids": ["41afee2a-9e5e-4481-9743-c36a0320e2d5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:17.654215", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:19.002 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqr2ipekdVFnHfRmYBTqmcN9MPmMp56Aqs'], u'uuids': [u'f9e29cab-df6c-4243-b5e3-24c9f51637f9']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.060112", "end": "2019-12-11 16:28:18.968307", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqr2ipekdVFnHfRmYBTqmcN9MPmMp56Aqs"], "labels": [], "masters": [], "uuids": ["f9e29cab-df6c-4243-b5e3-24c9f51637f9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:17.908195", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:20.110 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7Be4hUFVbCb1X5r4puo9RW2IZ0j40RVUe'], u'uuids': [u'7ab21ac3-2814-4ab8-ac1f-b2e270affc0a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.054364", "end": "2019-12-11 16:28:20.073577", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7Be4hUFVbCb1X5r4puo9RW2IZ0j40RVUe"], "labels": [], "masters": [], "uuids": ["7ab21ac3-2814-4ab8-ac1f-b2e270affc0a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:19.019213", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:20.165 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:20 +0000 (0:00:08.469) 0:04:29.767 **** 2019-12-11T16:28:20.207 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-11T16:28:20.210 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QDP6', u'wwn-0x5000c50091e0487b'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QDP6', u'wwn': u'0x5000c50091e0487b', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QDP6-part1', u'wwn-0x5000c50091e0487b-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QDP6", "wwn-0x5000c50091e0487b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QDP6-part1", "wwn-0x5000c50091e0487b-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QDP6", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-11T16:28:20.212 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50091e0487b"}}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:20.388 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqCihe1s3QSZqRWOf9XEEeYZJTHPycjuJg'], u'uuids': [u'44590309-0051-45bb-92eb-ed03dcb0aa93']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.071516", "end": "2019-12-11 16:28:20.345501", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqCihe1s3QSZqRWOf9XEEeYZJTHPycjuJg"], "labels": [], "masters": [], "uuids": ["44590309-0051-45bb-92eb-ed03dcb0aa93"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:19.273985", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition 2019-12-11T16:28:20.390 INFO:teuthology.task.ansible.out:the disk using fdisk or", "other utilities."]} 2019-12-11T16:28:20.437 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:20 +0000 (0:00:00.272) 0:04:30.039 **** 2019-12-11T16:28:20.485 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-11T16:28:20.486 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XPB7', u'wwn-0x5000c5009292944a'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XPB7', u'wwn': u'0x5000c5009292944a', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XPB7-part1', u'wwn-0x5000c5009292944a-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPB7", "wwn-0x5000c5009292944a"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPB7-part1", "wwn-0x5000c5009292944a-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XPB7", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-11T16:28:20.488 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009292944a"}}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:20.563 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-lhGyfM-85Tr-VXc9-nxok-ppxd-sfxn-n4qrPo', u'nvme-INTEL_SSDPEDMD400G4_PHFT6204013G400BGN', u'nvme-nvme.8086-50484654363230343031334734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.060483", "end": "2019-12-11 16:28:20.534997", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-lhGyfM-85Tr-VXc9-nxok-ppxd-sfxn-n4qrPo", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013G400BGN", "nvme-nvme.8086-50484654363230343031334734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:20.474514", "stderr": 2019-12-11T16:28:20.564 INFO:teuthology.task.ansible.out: "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0542421 s, 2.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0542421 s, 2.1 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:20.897 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-IysFJx-yV3m-EZ1g-Z2pD-rLXa-0KE9-ri31PU', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300AG400BGN', u'nvme-nvme.8086-43564654363233333030414734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.065376", "end": "2019-12-11 16:28:20.857788", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-IysFJx-yV3m-EZ1g-Z2pD-rLXa-0KE9-ri31PU", "nvme-INTEL_SSDPEDMD400G4_CVFT623300AG400BGN", "nvme-nvme.8086-43564654363233333030414734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:20.792412", "stderr": 2019-12-11T16:28:20.899 INFO:teuthology.task.ansible.out: "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0592537 s, 1.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0592537 s, 1.9 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:21.043 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7a0Uy2NHomSPuLZsOXgBeZmfV3UWF0N35'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.197578", "end": "2019-12-11 16:28:21.012708", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7a0Uy2NHomSPuLZsOXgBeZmfV3UWF0N35"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:20.815130", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.192575 s, 599 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.192575 s, 599 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:21.386 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqAR8rWIInwBkAd3DG1x8aHs50DEtcyRAK'], u'uuids': [u'4bf739d0-db1a-46b8-8d24-c01986f1a2a8']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.209655", "end": "2019-12-11 16:28:21.357329", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqAR8rWIInwBkAd3DG1x8aHs50DEtcyRAK"], "labels": [], "masters": [], "uuids": ["4bf739d0-db1a-46b8-8d24-c01986f1a2a8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:21.147674", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.203181 s, 568 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.203181 s, 568 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:21.571 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7iXZAxtLjtvvIPNN0znuBL3N0VZcYLyQK'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.210041", "end": "2019-12-11 16:28:21.531906", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7iXZAxtLjtvvIPNN0znuBL3N0VZcYLyQK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:21.321865", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.203284 s, 567 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.203284 s, 567 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:21.906 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqosTSy2AJ8JJVagTkFznEx4uN9m3gOywS'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.209258", "end": "2019-12-11 16:28:21.860640", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqosTSy2AJ8JJVagTkFznEx4uN9m3gOywS"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:21.651382", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.202404 s, 570 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.202404 s, 570 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:22.071 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk72KjxnfuUApEiqlg53cXqJhU2gSeEHjkB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.197523", "end": "2019-12-11 16:28:22.025797", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk72KjxnfuUApEiqlg53cXqJhU2gSeEHjkB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:21.828274", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.191435 s, 603 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.191435 s, 603 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:22.424 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqHInHXf61aC32VFcgf0khClYsP6dXYmAQ'], u'uuids': [u'32186ca3-28d8-417e-9dfb-fe44207f5704']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.202181", "end": "2019-12-11 16:28:22.390213", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqHInHXf61aC32VFcgf0khClYsP6dXYmAQ"], "labels": [], "masters": [], "uuids": ["32186ca3-28d8-417e-9dfb-fe44207f5704"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:22.188032", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.195822 s, 589 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.195822 s, 589 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:22.510 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7cKC1QCl0IbuFo9UbFn42Q2AZj9kKEwyj'], u'uuids': [u'41afee2a-9e5e-4481-9743-c36a0320e2d5']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.179108", "end": "2019-12-11 16:28:22.474022", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7cKC1QCl0IbuFo9UbFn42Q2AZj9kKEwyj"], "labels": [], "masters": [], "uuids": ["41afee2a-9e5e-4481-9743-c36a0320e2d5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:22.294914", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.172733 s, 668 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.172733 s, 668 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:22.886 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqr2ipekdVFnHfRmYBTqmcN9MPmMp56Aqs'], u'uuids': [u'f9e29cab-df6c-4243-b5e3-24c9f51637f9']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.184753", "end": "2019-12-11 16:28:22.848900", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqr2ipekdVFnHfRmYBTqmcN9MPmMp56Aqs"], "labels": [], "masters": [], "uuids": ["f9e29cab-df6c-4243-b5e3-24c9f51637f9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:22.664147", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.178564 s, 646 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.178564 s, 646 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:22.986 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7Be4hUFVbCb1X5r4puo9RW2IZ0j40RVUe'], u'uuids': [u'7ab21ac3-2814-4ab8-ac1f-b2e270affc0a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT6204013G400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.175341", "end": "2019-12-11 16:28:22.946102", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-fBeg0YmRQMC8YnzCvVHo9jzCxLZ4Ggk7Be4hUFVbCb1X5r4puo9RW2IZ0j40RVUe"], "labels": [], "masters": [], "uuids": ["7ab21ac3-2814-4ab8-ac1f-b2e270affc0a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT6204013G400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:22.770761", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.169123 s, 682 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.169123 s, 682 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:23.037 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:23 +0000 (0:00:02.599) 0:04:32.638 **** 2019-12-11T16:28:23.405 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqCihe1s3QSZqRWOf9XEEeYZJTHPycjuJg'], u'uuids': [u'44590309-0051-45bb-92eb-ed03dcb0aa93']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300AG400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.210247", "end": "2019-12-11 16:28:23.364165", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OFry4unVv3bol2evK1ShTwbB2fPgkaNqCihe1s3QSZqRWOf9XEEeYZJTHPycjuJg"], "labels": [], "masters": [], "uuids": ["44590309-0051-45bb-92eb-ed03dcb0aa93"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300AG400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-11 16:28:23.153918", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.203834 s, 566 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.203834 s, 566 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:23.456 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:23 +0000 (0:00:00.415) 0:04:33.054 **** 2019-12-11T16:28:23.464 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi079.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.095188", "end": "2019-12-11 16:28:23.417516", "rc": 0, "start": "2019-12-11 16:28:23.322328", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:23.497 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:23 +0000 (0:00:00.045) 0:04:33.099 **** 2019-12-11T16:28:23.849 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-12-11T16:28:23.850 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.050689", "end": "2019-12-11 16:28:23.818414", "rc": 0, "start": "2019-12-11 16:28:23.767725", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:23.886 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:23 +0000 (0:00:00.388) 0:04:33.487 **** 2019-12-11T16:28:23.892 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-11T16:28:23.893 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.080587", "end": "2019-12-11 16:28:23.822798", "rc": 0, "start": "2019-12-11 16:28:23.742211", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:23.935 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:23 +0000 (0:00:00.048) 0:04:33.536 **** 2019-12-11T16:28:23.940 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-11T16:28:23.982 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:23 +0000 (0:00:00.046) 0:04:33.583 **** 2019-12-11T16:28:24.019 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-12-11T16:28:24.021 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.060 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.078) 0:04:33.662 **** 2019-12-11T16:28:24.098 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-12-11T16:28:24.099 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.135 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.074) 0:04:33.736 **** 2019-12-11T16:28:24.162 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.202 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.067) 0:04:33.804 **** 2019-12-11T16:28:24.233 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-12-11T16:28:24.234 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.274 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.071) 0:04:33.875 **** 2019-12-11T16:28:24.290 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-12-11T16:28:24.292 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.055093", "end": "2019-12-11 16:28:24.263414", "rc": 0, "start": "2019-12-11 16:28:24.208321", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:24.341 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.062) 0:04:33.938 **** 2019-12-11T16:28:24.363 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-11T16:28:24.414 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.077) 0:04:34.016 **** 2019-12-11T16:28:24.441 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-12-11T16:28:24.443 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.488 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.074) 0:04:34.090 **** 2019-12-11T16:28:24.514 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-12-11T16:28:24.516 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.551 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.062) 0:04:34.153 **** 2019-12-11T16:28:24.577 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-11T16:28:24.579 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.579 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-12-11T16:28:24.581 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:28:24.615 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.064) 0:04:34.217 **** 2019-12-11T16:28:24.653 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.036) 0:04:34.253 **** 2019-12-11T16:28:24.668 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-11T16:28:24.669 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={u'block_used': 13037528, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227286956, u'size_available': 930967371776, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966955, u'device': u'/dev/sda1', u'inode_used': 88021, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227286956, "block_size": 4096, "block_total": 240324484, "block_used": 13037528, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966955, "inode_total": 61054976, "inode_used": 88021, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930967371776, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.707 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.053) 0:04:34.306 **** 2019-12-11T16:28:24.711 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-12-11T16:28:24.713 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.752 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.047) 0:04:34.354 **** 2019-12-11T16:28:24.758 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-11T16:28:24.759 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.798 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.045) 0:04:34.400 **** 2019-12-11T16:28:24.828 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-11T16:28:24.829 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.864 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.066) 0:04:34.466 **** 2019-12-11T16:28:24.901 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-11T16:28:24.903 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:24.943 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:24 +0000 (0:00:00.078) 0:04:34.545 **** 2019-12-11T16:28:24.945 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:24.981 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:24.984 INFO:teuthology.task.ansible.out:TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.022 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.078) 0:04:34.623 **** 2019-12-11T16:28:25.023 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:25.050 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:25.050 INFO:teuthology.task.ansible.out:TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-11T16:28:25.057 INFO:teuthology.task.ansible.out: skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi199.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-11T16:28:25.096 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.074) 0:04:34.697 **** 2019-12-11T16:28:25.129 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.033) 0:04:34.731 **** 2019-12-11T16:28:25.135 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-12-11T16:28:25.136 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.173 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.043) 0:04:34.775 **** 2019-12-11T16:28:25.179 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-11T16:28:25.180 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={u'block_used': 13037525, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227286959, u'size_available': 930967384064, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966955, u'device': u'/dev/sda1', u'inode_used': 88021, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227286959, "block_size": 4096, "block_total": 240324484, "block_used": 13037525, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966955, "inode_total": 61054976, "inode_used": 88021, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930967384064, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.219 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.045) 0:04:34.821 **** 2019-12-11T16:28:25.247 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-11T16:28:25.248 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.283 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.063) 0:04:34.884 **** 2019-12-11T16:28:25.309 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-11T16:28:25.310 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.347 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.064) 0:04:34.949 **** 2019-12-11T16:28:25.374 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-11T16:28:25.376 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.424 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.063) 0:04:35.012 **** 2019-12-11T16:28:25.448 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-12-11T16:28:25.451 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.486 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.074) 0:04:35.087 **** 2019-12-11T16:28:25.512 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-11T16:28:25.513 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.555 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.068) 0:04:35.156 **** 2019-12-11T16:28:25.581 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-12-11T16:28:25.582 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:25.618 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:25 +0000 (0:00:00.063) 0:04:35.220 **** 2019-12-11T16:28:25.981 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-12-11T16:28:25.983 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-11T16:28:26.031 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:26 +0000 (0:00:00.411) 0:04:35.631 **** 2019-12-11T16:28:26.165 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-11T16:28:26.210 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:26 +0000 (0:00:00.180) 0:04:35.812 **** 2019-12-11T16:28:26.810 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-11T16:28:26.812 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:26.822 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:27.264 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:27.312 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:27.726 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:27.759 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:28.176 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:28.197 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:28.657 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:28.701 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:28 +0000 (0:00:02.490) 0:04:38.302 **** 2019-12-11T16:28:28.709 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-11T16:28:28.753 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:28 +0000 (0:00:00.051) 0:04:38.354 **** 2019-12-11T16:28:29.002 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-11T16:28:29.003 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-11T16:28:29.047 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:29 +0000 (0:00:00.293) 0:04:38.648 **** 2019-12-11T16:28:29.064 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-11T16:28:29.111 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:29 +0000 (0:00:00.064) 0:04:38.713 **** 2019-12-11T16:28:29.119 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-11T16:28:29.119 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:29.152 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:29.412 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:29.455 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:29.713 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:29.746 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:30.021 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:30.045 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:30.328 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:30.376 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:30 +0000 (0:00:01.260) 0:04:39.974 **** 2019-12-11T16:28:30.386 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-11T16:28:30.432 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:30 +0000 (0:00:00.059) 0:04:40.033 **** 2019-12-11T16:28:30.679 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-11T16:28:30.727 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:30 +0000 (0:00:00.295) 0:04:40.329 **** 2019-12-11T16:28:30.760 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-11T16:28:30.802 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:30 +0000 (0:00:00.070) 0:04:40.399 **** 2019-12-11T16:28:30.804 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-12-11T16:28:30.805 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-11T16:28:30.852 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:30 +0000 (0:00:00.053) 0:04:40.453 **** 2019-12-11T16:28:30.859 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-11T16:28:30.900 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:30 +0000 (0:00:00.048) 0:04:40.501 **** 2019-12-11T16:28:31.560 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-12-11T16:28:31.570 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true} 2019-12-11T16:28:31.604 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:31 +0000 (0:00:00.702) 0:04:41.203 **** 2019-12-11T16:28:31.608 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true} 2019-12-11T16:28:31.651 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:31 +0000 (0:00:00.049) 0:04:41.253 **** 2019-12-11T16:28:32.334 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-12-11T16:28:32.335 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-11T16:28:32.372 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.720) 0:04:41.974 **** 2019-12-11T16:28:32.378 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-11T16:28:32.419 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.046) 0:04:42.020 **** 2019-12-11T16:28:32.425 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-12-11T16:28:32.425 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.464 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.045) 0:04:42.065 **** 2019-12-11T16:28:32.477 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.508 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.044) 0:04:42.110 **** 2019-12-11T16:28:32.516 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.567 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.058) 0:04:42.168 **** 2019-12-11T16:28:32.573 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.623 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.055) 0:04:42.224 **** 2019-12-11T16:28:32.631 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.671 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.048) 0:04:42.272 **** 2019-12-11T16:28:32.677 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.722 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.051) 0:04:42.323 **** 2019-12-11T16:28:32.727 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-12-11T16:28:32.728 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.769 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.047) 0:04:42.371 **** 2019-12-11T16:28:32.777 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.818 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.048) 0:04:42.420 **** 2019-12-11T16:28:32.823 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-12-11T16:28:32.824 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.864 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.046) 0:04:42.466 **** 2019-12-11T16:28:32.870 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.917 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.052) 0:04:42.518 **** 2019-12-11T16:28:32.923 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-12-11T16:28:32.924 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:32.963 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:32 +0000 (0:00:00.046) 0:04:42.565 **** 2019-12-11T16:28:32.969 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:33.008 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.044) 0:04:42.610 **** 2019-12-11T16:28:33.018 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:33.058 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.049) 0:04:42.659 **** 2019-12-11T16:28:33.064 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:33.107 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.048) 0:04:42.707 **** 2019-12-11T16:28:33.527 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-12-11T16:28:33.582 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.475) 0:04:43.183 **** 2019-12-11T16:28:33.599 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-12-11T16:28:33.644 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.061) 0:04:43.244 **** 2019-12-11T16:28:33.650 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-12-11T16:28:33.651 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:33.697 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.054) 0:04:43.298 **** 2019-12-11T16:28:33.703 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:33.740 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.043) 0:04:43.342 **** 2019-12-11T16:28:33.745 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-12-11T16:28:33.747 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:33.790 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.049) 0:04:43.392 **** 2019-12-11T16:28:33.799 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:33.837 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:33 +0000 (0:00:00.046) 0:04:43.438 **** 2019-12-11T16:28:34.714 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-12-11T16:28:34.714 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-12-11T16:28:34.757 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:34 +0000 (0:00:00.920) 0:04:44.359 **** 2019-12-11T16:28:34.763 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-12-11T16:28:34.810 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:34 +0000 (0:00:00.053) 0:04:44.412 **** 2019-12-11T16:28:34.817 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-12-11T16:28:34.818 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:34.858 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:34 +0000 (0:00:00.047) 0:04:44.459 **** 2019-12-11T16:28:34.863 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:34.904 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:34 +0000 (0:00:00.045) 0:04:44.504 **** 2019-12-11T16:28:35.323 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-12-11T16:28:35.325 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice sntp.service systemd-journald.socket syslog.target tmp.mount ntpdate.service basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "chronyd.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-12-11T16:28:35.326 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-12-11T16:28:35.328 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-11T16:28:35.384 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:35 +0000 (0:00:00.481) 0:04:44.986 **** 2019-12-11T16:28:35.426 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sntp.service ntpdate.service -.mount syslog.target tmp.mount systemd-journald.socket basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target chronyd.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-12-11T16:28:35.427 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-12-11T16:28:35.429 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-11T16:28:35.469 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:35 +0000 (0:00:00.084) 0:04:45.071 **** 2019-12-11T16:28:36.332 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-12-11T16:28:36.333 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-12-11T16:28:36.381 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:36 +0000 (0:00:00.908) 0:04:45.979 **** 2019-12-11T16:28:36.386 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-12-11T16:28:36.435 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:36 +0000 (0:00:00.057) 0:04:46.036 **** 2019-12-11T16:28:36.441 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-12-11T16:28:36.443 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:36.481 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:36 +0000 (0:00:00.045) 0:04:46.082 **** 2019-12-11T16:28:36.491 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:36.528 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:36 +0000 (0:00:00.046) 0:04:46.129 **** 2019-12-11T16:28:36.534 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2019-12-11T16:28:36.535 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:36.572 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:36 +0000 (0:00:00.044) 0:04:46.174 **** 2019-12-11T16:28:36.578 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:36.619 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:36 +0000 (0:00:00.046) 0:04:46.221 **** 2019-12-11T16:28:36.910 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-12-11T16:28:36.911 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-12-11T16:28:36.953 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:36 +0000 (0:00:00.333) 0:04:46.554 **** 2019-12-11T16:28:36.960 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-12-11T16:28:37.011 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:37 +0000 (0:00:00.058) 0:04:46.612 **** 2019-12-11T16:28:37.783 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-12-11T16:28:37.785 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-12-11T16:28:37.829 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:37 +0000 (0:00:00.818) 0:04:47.431 **** 2019-12-11T16:28:37.909 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-12-11T16:28:37.955 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:37 +0000 (0:00:00.122) 0:04:47.553 **** 2019-12-11T16:28:39.622 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.3.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.3.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-11T16:28:39.678 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:39 +0000 (0:00:01.725) 0:04:49.279 **** 2019-12-11T16:28:39.686 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.3.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.3.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-11T16:28:39.740 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:39 +0000 (0:00:00.062) 0:04:49.341 **** 2019-12-11T16:28:39.744 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-12-11T16:28:39.746 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:39.816 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:39 +0000 (0:00:00.069) 0:04:49.411 **** 2019-12-11T16:28:39.820 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:39.878 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:39 +0000 (0:00:00.068) 0:04:49.479 **** 2019-12-11T16:28:39.888 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-12-11T16:28:39.891 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-12-11T16:28:39.951 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:39 +0000 (0:00:00.073) 0:04:49.552 **** 2019-12-11T16:28:39.964 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-12-11T16:28:40.024 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.073) 0:04:49.626 **** 2019-12-11T16:28:40.262 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-12-11T16:28:40.263 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-12-11T16:28:40.316 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.292) 0:04:49.918 **** 2019-12-11T16:28:40.355 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:40.356 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi079.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-12-11T16:28:40.408 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.086) 0:04:50.004 **** 2019-12-11T16:28:40.440 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.035) 0:04:50.040 **** 2019-12-11T16:28:40.443 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-12-11T16:28:40.445 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:40.489 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.045) 0:04:50.086 **** 2019-12-11T16:28:40.492 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi199.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-11T16:28:40.531 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.045) 0:04:50.132 **** 2019-12-11T16:28:40.539 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-11T16:28:40.585 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.046) 0:04:50.178 **** TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:40.624 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.047) 0:04:50.226 **** 2019-12-11T16:28:40.634 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:40.674 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.048) 0:04:50.274 **** 2019-12-11T16:28:40.928 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-12-11T16:28:40.929 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-11T16:28:40.980 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:40 +0000 (0:00:00.307) 0:04:50.582 **** 2019-12-11T16:28:40.982 INFO:teuthology.task.ansible.out: 2019-12-11T16:28:41.013 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-11T16:28:41.049 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.067) 0:04:50.650 **** 2019-12-11T16:28:41.288 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi199.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-11T16:28:41.328 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.278) 0:04:50.928 **** 2019-12-11T16:28:41.353 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-12-11T16:28:41.355 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.396 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.069) 0:04:50.997 **** 2019-12-11T16:28:41.405 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi079.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-11T16:28:41.460 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.064) 0:04:51.062 **** 2019-12-11T16:28:41.468 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi199.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.473 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.486 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.487 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.533 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.072) 0:04:51.134 **** 2019-12-11T16:28:41.548 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.591 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.057) 0:04:51.192 **** 2019-12-11T16:28:41.601 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi199.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.622 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi199.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.663 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.072) 0:04:51.264 **** 2019-12-11T16:28:41.668 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-11T16:28:41.669 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.670 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.672 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.714 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.047) 0:04:51.311 **** 2019-12-11T16:28:41.717 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.782 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.072) 0:04:51.383 **** 2019-12-11T16:28:41.791 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi079.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.797 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi079.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.841 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.059) 0:04:51.443 **** 2019-12-11T16:28:41.848 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-12-11T16:28:41.849 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.892 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.050) 0:04:51.493 **** 2019-12-11T16:28:41.901 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-11T16:28:41.903 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.946 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.051) 0:04:51.545 **** 2019-12-11T16:28:41.955 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:41.999 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:41 +0000 (0:00:00.054) 0:04:51.600 **** 2019-12-11T16:28:42.004 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-12-11T16:28:42.013 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.065 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.066) 0:04:51.667 **** 2019-12-11T16:28:42.073 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.113 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.045) 0:04:51.713 **** 2019-12-11T16:28:42.117 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-11T16:28:42.119 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.161 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.049) 0:04:51.762 **** 2019-12-11T16:28:42.167 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-12-11T16:28:42.169 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.208 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.046) 0:04:51.809 **** 2019-12-11T16:28:42.213 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-12-11T16:28:42.214 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.255 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.045) 0:04:51.855 **** 2019-12-11T16:28:42.268 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.306 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.051) 0:04:51.907 **** 2019-12-11T16:28:42.314 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.364 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.057) 0:04:51.964 **** 2019-12-11T16:28:42.369 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-12-11T16:28:42.371 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.410 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.047) 0:04:52.012 **** 2019-12-11T16:28:42.416 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-11T16:28:42.418 INFO:teuthology.task.ansible.out:skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.458 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.047) 0:04:52.059 **** 2019-12-11T16:28:42.464 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-11T16:28:42.466 INFO:teuthology.task.ansible.out:skipping: [smithi199.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.503 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.045) 0:04:52.105 **** 2019-12-11T16:28:42.511 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.553 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.048) 0:04:52.153 **** 2019-12-11T16:28:42.582 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi079.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-11T16:28:42.622 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:42 +0000 (0:00:00.069) 0:04:52.223 **** 2019-12-11T16:28:42.880 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi199.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.013518", "end": "2019-12-11 16:28:42.820684", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-11 16:28:42.807166", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.014 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.016502", "end": "2019-12-11 16:28:42.954769", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-11 16:28:42.938267", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.175 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.007197", "end": "2019-12-11 16:28:43.137243", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-11 16:28:43.130046", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.316 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.007301", "end": "2019-12-11 16:28:43.278263", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-11 16:28:43.270962", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.473 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.006136", "end": "2019-12-11 16:28:43.427648", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-11 16:28:43.421512", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.629 INFO:teuthology.task.ansible.out:ok: [smithi079.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.007595", "end": "2019-12-11 16:28:43.584639", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-11 16:28:43.577044", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.781 INFO:teuthology.task.ansible.out:ok: [smithi199.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007396", "end": "2019-12-11 16:28:43.739388", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-11 16:28:43.731992", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.831 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:43 +0000 (0:00:01.209) 0:04:53.432 **** 2019-12-11T16:28:43.857 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-11T16:28:43.900 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:43 +0000 (0:00:00.068) 0:04:53.501 **** 2019-12-11T16:28:43.943 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi079.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.008324", "end": "2019-12-11 16:28:43.903585", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-11 16:28:43.895261", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-11T16:28:43.991 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:43 +0000 (0:00:00.086) 0:04:53.588 **** 2019-12-11T16:28:44.014 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-11T16:28:44.057 INFO:teuthology.task.ansible.out:Wednesday 11 December 2019 16:28:44 +0000 (0:00:00.065) 0:04:53.654 **** 2019-12-11T16:28:44.235 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-12-11T16:28:44.235 INFO:teuthology.task.ansible.out:changed: [smithi199.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-11T16:28:44.383 INFO:teuthology.task.ansible.out:changed: [smithi079.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-11T16:28:44.476 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2019-12-11T16:28:44.478 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-12-11T16:28:44.482 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi079.front.sepia.ceph.com : ok=121 changed=41 unreachable=0 failed=0 skipped=160 rescued=0 ignored=3 smithi199.front.sepia.ceph.com : ok=121 changed=41 unreachable=0 failed=0 skipped=160 rescued=0 ignored=3 Wednesday 11 December 2019 16:28:44 +0000 (0:00:00.430) 0:04:54.084 **** =============================================================================== 2019-12-11T16:28:44.486 INFO:teuthology.task.ansible.out:users : Create all admin users with sudo access. ----------------------- 36.98s users : Update authorized_keys using the keys repo --------------------- 35.67s common : nrpe - Ensure SELinux policy is up to date -------------------- 26.02s testnode : Install packages -------------------------------------------- 18.13s users : Remove revoked users ------------------------------------------- 15.95s common : Update yum cache ---------------------------------------------- 13.28s testnode : Zap all non-root disks --------------------------------------- 8.47s testnode : Ensure ceph packages are not present. ------------------------ 7.69s common : nrpe - Remove SELinux policy package --------------------------- 7.35s common : nrpe - Load SELinux policy package ----------------------------- 7.08s common : Make sure smartmontools is installed --------------------------- 3.59s Gathering Facts --------------------------------------------------------- 3.41s testnode : Ensure ceph dependency packages are not present. ------------- 3.02s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 2.99s ansible-managed : Add authorized keys for the ansible user. ------------- 2.96s testnode : Blow away lingering OSD data and FSIDs ----------------------- 2.60s testnode : Create logical volume(s) ------------------------------------- 2.49s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 2.36s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 2.30s users : Update authorized_keys for each user with literal keys ---------- 1.94s 2019-12-11T16:28:44.755 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi199.front.sepia.ceph.com')] 2019-12-11T16:28:44.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:28:44.759 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi079.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:28:44.866 INFO:teuthology.orchestra.run.smithi079:> true 2019-12-11T16:28:45.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-11T16:28:45.257 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi199.front.sepia.ceph.com', 'timeout': 60} 2019-12-11T16:28:45.367 INFO:teuthology.orchestra.run.smithi199:> true 2019-12-11T16:28:45.750 INFO:teuthology.run_tasks:Running task clock... 2019-12-11T16:28:45.795 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-12-11T16:28:45.795 INFO:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-11T16:28:45.933 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-12-11T16:28:56.012 INFO:teuthology.orchestra.run.smithi079.stdout:ntpd: time slew +0.000348s 2019-12-11T16:28:56.049 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit not found. 2019-12-11T16:28:56.128 INFO:teuthology.orchestra.run.smithi079.stdout: remote refid st t when poll reach delay offset jitter 2019-12-11T16:28:56.128 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================== 2019-12-11T16:28:56.128 INFO:teuthology.orchestra.run.smithi079.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:28:56.129 INFO:teuthology.orchestra.run.smithi079.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:28:56.129 INFO:teuthology.orchestra.run.smithi079.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:28:56.129 INFO:teuthology.orchestra.run.smithi079.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:28:56.130 INFO:teuthology.orchestra.run.smithi199:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-11T16:28:56.268 INFO:teuthology.orchestra.run.smithi199.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-12-11T16:29:06.347 INFO:teuthology.orchestra.run.smithi199.stdout:ntpd: time slew -0.001212s 2019-12-11T16:29:06.385 INFO:teuthology.orchestra.run.smithi199.stderr:Failed to start ntp.service: Unit not found. 2019-12-11T16:29:06.474 INFO:teuthology.orchestra.run.smithi199.stdout: remote refid st t when poll reach delay offset jitter 2019-12-11T16:29:06.474 INFO:teuthology.orchestra.run.smithi199.stdout:============================================================================== 2019-12-11T16:29:06.474 INFO:teuthology.orchestra.run.smithi199.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:29:06.474 INFO:teuthology.orchestra.run.smithi199.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:29:06.475 INFO:teuthology.orchestra.run.smithi199.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:29:06.475 INFO:teuthology.orchestra.run.smithi199.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-11T16:29:06.475 INFO:teuthology.run_tasks:Running task install... 2019-12-11T16:29:06.509 DEBUG:teuthology.task.install:project ceph 2019-12-11T16:29:06.509 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'notcmalloc', 'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518'}} 2019-12-11T16:29:06.509 DEBUG:teuthology.task.install:config {'flavor': 'notcmalloc', 'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518'} 2019-12-11T16:29:06.509 INFO:teuthology.task.install:Using flavor: notcmalloc 2019-12-11T16:29:06.514 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-12-11T16:29:06.514 INFO:teuthology.task.install:extra packages: [] 2019-12-11T16:29:06.514 INFO:teuthology.orchestra.run.smithi079:> sudo lsb_release -is 2019-12-11T16:29:06.681 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS 2019-12-11T16:29:06.682 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-11T16:29:06.682 INFO:teuthology.orchestra.run.smithi199:> sudo lsb_release -is 2019-12-11T16:29:06.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'notcmalloc', 'extras': None} 2019-12-11T16:29:06.683 INFO:teuthology.orchestra.run.smithi079:> sudo lsb_release -is 2019-12-11T16:29:06.821 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS 2019-12-11T16:29:06.823 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-11T16:29:06.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=1b78fd04fb1980223b57f0a3f644592e4e26b518 2019-12-11T16:29:06.842 INFO:teuthology.orchestra.run.smithi199.stdout:CentOS 2019-12-11T16:29:06.844 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-11T16:29:06.844 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'notcmalloc', 'extras': None} 2019-12-11T16:29:06.844 INFO:teuthology.orchestra.run.smithi199:> sudo lsb_release -is 2019-12-11T16:29:06.988 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-abhi-testing-2019-12-10-1759/1b78fd04fb1980223b57f0a3f644592e4e26b518/centos/7/flavors/notcmalloc/ 2019-12-11T16:29:06.988 INFO:teuthology.task.install.rpm:Package version is 15.0.0-7938.g1b78fd0 2019-12-11T16:29:06.998 INFO:teuthology.orchestra.run.smithi199.stdout:CentOS 2019-12-11T16:29:06.998 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-11T16:29:06.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=1b78fd04fb1980223b57f0a3f644592e4e26b518 2019-12-11T16:29:07.117 INFO:teuthology.orchestra.run.smithi079:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-12-11T16:29:07.165 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-abhi-testing-2019-12-10-1759/1b78fd04fb1980223b57f0a3f644592e4e26b518/centos/7/flavors/notcmalloc/ 2019-12-11T16:29:07.165 INFO:teuthology.task.install.rpm:Package version is 15.0.0-7938.g1b78fd0 2019-12-11T16:29:07.236 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2019-12-11T16:29:07.236 INFO:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b78fd04fb1980223b57f0a3f644592e4e26b518/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-12-11T16:29:07.283 INFO:teuthology.orchestra.run.smithi199:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-12-11T16:29:07.345 INFO:teuthology.orchestra.run.smithi079:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-12-11T16:29:07.401 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2019-12-11T16:29:07.401 INFO:teuthology.orchestra.run.smithi199:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b78fd04fb1980223b57f0a3f644592e4e26b518/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-12-11T16:29:07.459 INFO:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-12-11T16:29:07.504 INFO:teuthology.orchestra.run.smithi199:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-12-11T16:29:07.562 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2019-12-11T16:29:07.565 INFO:teuthology.orchestra.run.smithi079:> sudo yum clean all 2019-12-11T16:29:07.586 INFO:teuthology.orchestra.run.smithi199:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-12-11T16:29:07.655 INFO:teuthology.orchestra.run.smithi199.stdout:check_obsoletes = 1 2019-12-11T16:29:07.657 INFO:teuthology.orchestra.run.smithi199:> sudo yum clean all 2019-12-11T16:29:07.766 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:29:07.806 INFO:teuthology.orchestra.run.smithi079.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-12-11T16:29:07.806 INFO:teuthology.orchestra.run.smithi079.stdout: : lab-extras updates 2019-12-11T16:29:07.816 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:29:07.854 INFO:teuthology.orchestra.run.smithi199.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-12-11T16:29:07.854 INFO:teuthology.orchestra.run.smithi199.stdout: : lab-extras updates 2019-12-11T16:29:07.854 INFO:teuthology.orchestra.run.smithi079.stdout:Cleaning up list of fastest mirrors 2019-12-11T16:29:07.855 INFO:teuthology.orchestra.run.smithi079.stdout:Other repos take up 1.5 M of disk space (use --verbose for details) 2019-12-11T16:29:07.865 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2019-12-11T16:29:07.900 INFO:teuthology.orchestra.run.smithi199.stdout:Cleaning up list of fastest mirrors 2019-12-11T16:29:07.900 INFO:teuthology.orchestra.run.smithi199.stdout:Other repos take up 1.5 M of disk space (use --verbose for details) 2019-12-11T16:29:07.913 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-radosgw 2019-12-11T16:29:08.022 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:29:08.059 INFO:teuthology.orchestra.run.smithi079.stdout:Determining fastest mirrors 2019-12-11T16:29:08.165 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:29:08.209 INFO:teuthology.orchestra.run.smithi199.stdout:Determining fastest mirrors 2019-12-11T16:29:08.696 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:29:08.697 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:29:08.698 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:29:08.698 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:29:08.855 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:29:08.856 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:29:08.856 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:29:08.857 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:29:19.243 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:29:19.595 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:29:19.595 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:29:19.596 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.608 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.714 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.714 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: librados2 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.716 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.718 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.718 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.720 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libceph-common.so.2()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.721 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.722 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.723 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:29:19.723 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-base.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.732 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libcephfs2 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.733 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.734 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: librbd1 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.735 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.738 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.738 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.740 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-12-11T16:29:19.741 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-12-11T16:29:19.741 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-12-11T16:29:19.742 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librados2.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be an update 2019-12-11T16:29:19.743 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2019-12-11T16:29:19.744 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librgw2.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.745 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:29:19.745 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.750 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.751 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.752 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libradosstriper1 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.753 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-rados = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.754 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.755 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.756 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.756 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.757 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.758 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.759 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:19.759 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-12-11T16:29:19.759 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libcephfs2.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.760 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-12-11T16:29:19.760 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librbd1.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be an update 2019-12-11T16:29:19.761 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-12-11T16:29:19.762 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-11T16:29:19.762 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-11T16:29:19.763 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:29:19.763 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-12-11T16:29:19.764 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.765 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.765 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.765 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-cephfs.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.766 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-12-11T16:29:19.766 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-rados.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.767 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-rbd.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.767 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-rgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:19.767 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-12-11T16:29:20.080 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:29:20.433 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:29:20.434 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:29:20.434 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.447 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.553 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.553 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: librados2 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.555 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.557 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.557 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.559 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libceph-common.so.2()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.560 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.561 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.562 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:29:20.562 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-base.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.570 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libcephfs2 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.572 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.573 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: librbd1 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.574 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.575 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.576 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.579 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-12-11T16:29:20.579 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-12-11T16:29:20.580 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-12-11T16:29:20.580 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librados2.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be an update 2019-12-11T16:29:20.582 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2019-12-11T16:29:20.583 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librgw2.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.584 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:29:20.584 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.589 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.590 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.591 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libradosstriper1 = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.592 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-rados = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.593 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.594 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.595 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.595 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.596 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.597 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.598 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:20.598 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-12-11T16:29:20.598 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package libcephfs2.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.599 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-12-11T16:29:20.599 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librbd1.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be an update 2019-12-11T16:29:20.600 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-12-11T16:29:20.600 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-11T16:29:20.601 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-11T16:29:20.601 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:29:20.602 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-12-11T16:29:20.603 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.604 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.604 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.604 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-cephfs.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.605 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-12-11T16:29:20.605 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-rados.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.606 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-rbd.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.606 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-rgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:20.606 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-12-11T16:29:20.624 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:29:20.709 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:20.710 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:29:20.713 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:20.714 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:29:20.714 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository 2019-12-11T16:29:20.714 INFO:teuthology.orchestra.run.smithi079.stdout: Size 2019-12-11T16:29:20.714 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:29:20.714 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:29:20.714 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 6.3 M 2019-12-11T16:29:20.714 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:29:20.715 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 5.6 M 2019-12-11T16:29:20.715 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 20 M 2019-12-11T16:29:20.715 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 20 k 2019-12-11T16:29:20.715 INFO:teuthology.orchestra.run.smithi079.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-12-11T16:29:20.715 INFO:teuthology.orchestra.run.smithi079.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-12-11T16:29:20.715 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 490 k 2019-12-11T16:29:20.716 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-12-11T16:29:20.716 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-12-11T16:29:20.717 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 374 k 2019-12-11T16:29:20.717 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2019-12-11T16:29:20.717 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 5.5 M 2019-12-11T16:29:20.717 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-12-11T16:29:20.717 INFO:teuthology.orchestra.run.smithi079.stdout: python-ceph-argparse x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 35 k 2019-12-11T16:29:20.717 INFO:teuthology.orchestra.run.smithi079.stdout: python-ceph-common x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 42 k 2019-12-11T16:29:20.718 INFO:teuthology.orchestra.run.smithi079.stdout: python-cephfs x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 102 k 2019-12-11T16:29:20.718 INFO:teuthology.orchestra.run.smithi079.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-12-11T16:29:20.718 INFO:teuthology.orchestra.run.smithi079.stdout: python-rados x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 209 k 2019-12-11T16:29:20.718 INFO:teuthology.orchestra.run.smithi079.stdout: python-rbd x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 199 k 2019-12-11T16:29:20.718 INFO:teuthology.orchestra.run.smithi079.stdout: python-rgw x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 77 k 2019-12-11T16:29:20.718 INFO:teuthology.orchestra.run.smithi079.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-12-11T16:29:20.718 INFO:teuthology.orchestra.run.smithi079.stdout:Updating for dependencies: 2019-12-11T16:29:20.719 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 3.5 M 2019-12-11T16:29:20.719 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 1.8 M 2019-12-11T16:29:20.719 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:20.719 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:29:20.719 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:29:20.719 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+20 Dependent packages) 2019-12-11T16:29:20.719 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade ( 2 Dependent packages) 2019-12-11T16:29:20.720 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:20.720 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 45 M 2019-12-11T16:29:20.720 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:29:20.721 INFO:teuthology.orchestra.run.smithi079.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-11T16:29:21.469 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:29:21.556 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:21.556 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:29:21.559 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:21.559 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:29:21.559 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository 2019-12-11T16:29:21.559 INFO:teuthology.orchestra.run.smithi199.stdout: Size 2019-12-11T16:29:21.560 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:29:21.560 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:29:21.560 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-radosgw x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 6.3 M 2019-12-11T16:29:21.560 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:29:21.560 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-base x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 5.6 M 2019-12-11T16:29:21.560 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-common x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 20 M 2019-12-11T16:29:21.561 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-selinux x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 20 k 2019-12-11T16:29:21.561 INFO:teuthology.orchestra.run.smithi199.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-12-11T16:29:21.561 INFO:teuthology.orchestra.run.smithi199.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-12-11T16:29:21.561 INFO:teuthology.orchestra.run.smithi199.stdout: libcephfs2 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 490 k 2019-12-11T16:29:21.561 INFO:teuthology.orchestra.run.smithi199.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-12-11T16:29:21.561 INFO:teuthology.orchestra.run.smithi199.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-12-11T16:29:21.562 INFO:teuthology.orchestra.run.smithi199.stdout: libradosstriper1 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 374 k 2019-12-11T16:29:21.562 INFO:teuthology.orchestra.run.smithi199.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2019-12-11T16:29:21.562 INFO:teuthology.orchestra.run.smithi199.stdout: librgw2 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 5.5 M 2019-12-11T16:29:21.562 INFO:teuthology.orchestra.run.smithi199.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-12-11T16:29:21.562 INFO:teuthology.orchestra.run.smithi199.stdout: python-ceph-argparse x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 35 k 2019-12-11T16:29:21.562 INFO:teuthology.orchestra.run.smithi199.stdout: python-ceph-common x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 42 k 2019-12-11T16:29:21.562 INFO:teuthology.orchestra.run.smithi199.stdout: python-cephfs x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 102 k 2019-12-11T16:29:21.563 INFO:teuthology.orchestra.run.smithi199.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-12-11T16:29:21.563 INFO:teuthology.orchestra.run.smithi199.stdout: python-rados x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 209 k 2019-12-11T16:29:21.563 INFO:teuthology.orchestra.run.smithi199.stdout: python-rbd x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 199 k 2019-12-11T16:29:21.563 INFO:teuthology.orchestra.run.smithi199.stdout: python-rgw x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 77 k 2019-12-11T16:29:21.563 INFO:teuthology.orchestra.run.smithi199.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-12-11T16:29:21.563 INFO:teuthology.orchestra.run.smithi199.stdout:Updating for dependencies: 2019-12-11T16:29:21.563 INFO:teuthology.orchestra.run.smithi199.stdout: librados2 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 3.5 M 2019-12-11T16:29:21.564 INFO:teuthology.orchestra.run.smithi199.stdout: librbd1 x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 1.8 M 2019-12-11T16:29:21.564 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:21.564 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:29:21.564 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:29:21.564 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+20 Dependent packages) 2019-12-11T16:29:21.564 INFO:teuthology.orchestra.run.smithi199.stdout:Upgrade ( 2 Dependent packages) 2019-12-11T16:29:21.565 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:21.565 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 45 M 2019-12-11T16:29:21.565 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:29:21.565 INFO:teuthology.orchestra.run.smithi199.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-11T16:29:24.462 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:29:24.462 INFO:teuthology.orchestra.run.smithi079.stdout:Total 12 MB/s | 45 MB 00:03 2019-12-11T16:29:24.479 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:29:24.556 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:29:24.789 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:29:24.789 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:29:24.891 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:29:24.891 INFO:teuthology.orchestra.run.smithi199.stdout:Total 13 MB/s | 45 MB 00:03 2019-12-11T16:29:24.908 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:29:24.983 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:29:25.199 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:29:25.199 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:29:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/25 2019-12-11T16:29:25.864 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/25 2019-12-11T16:29:25.884 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/25 2019-12-11T16:29:26.166 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 3/25 2019-12-11T16:29:26.220 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/25 2019-12-11T16:29:26.466 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:python-ceph-argparse-15.0.0-7938.g1b78fd0.el7.x86_64 4/25 2019-12-11T16:29:26.505 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 3/25 2019-12-11T16:29:26.776 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 5/25 2019-12-11T16:29:26.796 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:python-ceph-argparse-15.0.0-7938.g1b78fd0.el7.x86_64 4/25 2019-12-11T16:29:27.104 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 5/25 2019-12-11T16:29:27.149 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 6/25 2019-12-11T16:29:27.474 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 6/25 2019-12-11T16:29:27.922 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 7/25 2019-12-11T16:29:28.228 INFO:teuthology.orchestra.run.smithi079.stdout: Updating : 2:librados2-15.0.0-7938.g1b78fd0.el7.x86_64 8/25 2019-12-11T16:29:28.264 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 7/25 2019-12-11T16:29:28.595 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:python-rados-15.0.0-7938.g1b78fd0.el7.x86_64 9/25 2019-12-11T16:29:28.610 INFO:teuthology.orchestra.run.smithi199.stdout: Updating : 2:librados2-15.0.0-7938.g1b78fd0.el7.x86_64 8/25 2019-12-11T16:29:28.929 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:python-rados-15.0.0-7938.g1b78fd0.el7.x86_64 9/25 2019-12-11T16:29:29.130 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:libcephfs2-15.0.0-7938.g1b78fd0.el7.x86_64 10/25 2019-12-11T16:29:29.426 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:libcephfs2-15.0.0-7938.g1b78fd0.el7.x86_64 10/25 2019-12-11T16:29:30.059 INFO:teuthology.orchestra.run.smithi079.stdout: Updating : 2:librbd1-15.0.0-7938.g1b78fd0.el7.x86_64 11/25 2019-12-11T16:29:30.312 INFO:teuthology.orchestra.run.smithi199.stdout: Updating : 2:librbd1-15.0.0-7938.g1b78fd0.el7.x86_64 11/25 2019-12-11T16:29:30.386 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:librgw2-15.0.0-7938.g1b78fd0.el7.x86_64 12/25 2019-12-11T16:29:30.627 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:librgw2-15.0.0-7938.g1b78fd0.el7.x86_64 12/25 2019-12-11T16:29:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:python-rgw-15.0.0-7938.g1b78fd0.el7.x86_64 13/25 2019-12-11T16:29:30.930 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:python-rgw-15.0.0-7938.g1b78fd0.el7.x86_64 13/25 2019-12-11T16:29:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:python-rbd-15.0.0-7938.g1b78fd0.el7.x86_64 14/25 2019-12-11T16:29:31.210 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:python-rbd-15.0.0-7938.g1b78fd0.el7.x86_64 14/25 2019-12-11T16:29:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:python-cephfs-15.0.0-7938.g1b78fd0.el7.x86_64 15/25 2019-12-11T16:29:31.557 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:python-cephfs-15.0.0-7938.g1b78fd0.el7.x86_64 15/25 2019-12-11T16:29:31.671 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:libradosstriper1-15.0.0-7938.g1b78fd0.el7.x86_64 16/25 2019-12-11T16:29:31.876 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:libradosstriper1-15.0.0-7938.g1b78fd0.el7.x86_64 16/25 2019-12-11T16:29:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:python-ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 17/25 2019-12-11T16:29:32.101 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:python-ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 17/25 2019-12-11T16:29:32.226 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 18/25 2019-12-11T16:29:32.373 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 18/25 2019-12-11T16:29:34.648 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 19/25 2019-12-11T16:29:34.779 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 19/25 2019-12-11T16:29:35.835 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 20/25 2019-12-11T16:29:35.942 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 20/25 2019-12-11T16:29:36.382 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 21/25 2019-12-11T16:29:36.510 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 21/25 2019-12-11T16:29:45.866 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-selinux-15.0.0-7938.g1b78fd0.el7.x86_64 22/25 2019-12-11T16:29:46.079 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-selinux-15.0.0-7938.g1b78fd0.el7.x86_64 22/25 2019-12-11T16:29:46.223 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 23/25 2019-12-11T16:29:46.475 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-11T16:29:47.149 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 23/25 2019-12-11T16:29:47.534 INFO:teuthology.orchestra.run.smithi199.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-11T16:29:47.782 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-11T16:29:47.815 INFO:teuthology.orchestra.run.smithi199.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-11T16:29:47.878 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:python-rados-15.0.0-7938.g1b78fd0.el7.x86_64 1/25 2019-12-11T16:29:47.919 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:python-rados-15.0.0-7938.g1b78fd0.el7.x86_64 1/25 2019-12-11T16:29:47.978 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2/25 2019-12-11T16:29:47.994 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-base-15.0.0-7938.g1b78fd0.el7.x86_64 2/25 2019-12-11T16:29:48.111 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 3/25 2019-12-11T16:29:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 3/25 2019-12-11T16:29:48.195 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 4/25 2019-12-11T16:29:48.271 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 4/25 2019-12-11T16:29:48.278 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:libcephfs2-15.0.0-7938.g1b78fd0.el7.x86_64 5/25 2019-12-11T16:29:48.372 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:libcephfs2-15.0.0-7938.g1b78fd0.el7.x86_64 5/25 2019-12-11T16:29:48.429 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:python-rbd-15.0.0-7938.g1b78fd0.el7.x86_64 6/25 2019-12-11T16:29:48.497 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:python-rbd-15.0.0-7938.g1b78fd0.el7.x86_64 6/25 2019-12-11T16:29:48.554 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 7/25 2019-12-11T16:29:48.614 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 7/25 2019-12-11T16:29:48.680 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 8/25 2019-12-11T16:29:48.757 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 8/25 2019-12-11T16:29:48.789 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:librados2-15.0.0-7938.g1b78fd0.el7.x86_64 9/25 2019-12-11T16:29:48.874 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:librados2-15.0.0-7938.g1b78fd0.el7.x86_64 9/25 2019-12-11T16:29:48.906 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-selinux-15.0.0-7938.g1b78fd0.el7.x86_64 10/25 2019-12-11T16:29:48.974 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-selinux-15.0.0-7938.g1b78fd0.el7.x86_64 10/25 2019-12-11T16:29:49.023 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:python-cephfs-15.0.0-7938.g1b78fd0.el7.x86_64 11/25 2019-12-11T16:29:49.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:python-cephfs-15.0.0-7938.g1b78fd0.el7.x86_64 11/25 2019-12-11T16:29:49.158 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:python-rgw-15.0.0-7938.g1b78fd0.el7.x86_64 12/25 2019-12-11T16:29:49.237 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:python-rgw-15.0.0-7938.g1b78fd0.el7.x86_64 12/25 2019-12-11T16:29:49.272 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 13/25 2019-12-11T16:29:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 13/25 2019-12-11T16:29:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 14/25 2019-12-11T16:29:49.425 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 14/25 2019-12-11T16:29:49.477 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:librbd1-15.0.0-7938.g1b78fd0.el7.x86_64 15/25 2019-12-11T16:29:49.526 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:librbd1-15.0.0-7938.g1b78fd0.el7.x86_64 15/25 2019-12-11T16:29:49.560 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:python-ceph-argparse-15.0.0-7938.g1b78fd0.el7.x86_64 16/25 2019-12-11T16:29:49.609 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:python-ceph-argparse-15.0.0-7938.g1b78fd0.el7.x86_64 16/25 2019-12-11T16:29:49.653 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 17/25 2019-12-11T16:29:49.718 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-radosgw-15.0.0-7938.g1b78fd0.el7.x86_64 17/25 2019-12-11T16:29:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:libradosstriper1-15.0.0-7938.g1b78fd0.el7.x86_64 18/25 2019-12-11T16:29:49.828 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:libradosstriper1-15.0.0-7938.g1b78fd0.el7.x86_64 18/25 2019-12-11T16:29:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 19/25 2019-12-11T16:29:49.927 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 19/25 2019-12-11T16:29:50.019 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:python-ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 20/25 2019-12-11T16:29:50.046 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:python-ceph-common-15.0.0-7938.g1b78fd0.el7.x86_64 20/25 2019-12-11T16:29:50.120 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 21/25 2019-12-11T16:29:50.164 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 21/25 2019-12-11T16:29:50.224 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 22/25 2019-12-11T16:29:50.289 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 22/25 2019-12-11T16:29:50.371 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:librgw2-15.0.0-7938.g1b78fd0.el7.x86_64 23/25 2019-12-11T16:29:50.372 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-11T16:29:50.408 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:librgw2-15.0.0-7938.g1b78fd0.el7.x86_64 23/25 2019-12-11T16:29:50.408 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-11T16:29:50.761 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-11T16:29:50.761 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:50.762 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:29:50.762 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-radosgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.762 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:50.762 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:29:50.762 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-base.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.762 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.763 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-selinux.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.763 INFO:teuthology.orchestra.run.smithi199.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-12-11T16:29:50.763 INFO:teuthology.orchestra.run.smithi199.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-12-11T16:29:50.763 INFO:teuthology.orchestra.run.smithi199.stdout: libcephfs2.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.763 INFO:teuthology.orchestra.run.smithi199.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-12-11T16:29:50.763 INFO:teuthology.orchestra.run.smithi199.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-12-11T16:29:50.763 INFO:teuthology.orchestra.run.smithi199.stdout: libradosstriper1.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.764 INFO:teuthology.orchestra.run.smithi199.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2019-12-11T16:29:50.764 INFO:teuthology.orchestra.run.smithi199.stdout: librgw2.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.764 INFO:teuthology.orchestra.run.smithi199.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-12-11T16:29:50.764 INFO:teuthology.orchestra.run.smithi199.stdout: python-ceph-argparse.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.764 INFO:teuthology.orchestra.run.smithi199.stdout: python-ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.764 INFO:teuthology.orchestra.run.smithi199.stdout: python-cephfs.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.764 INFO:teuthology.orchestra.run.smithi199.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout: python-rados.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout: python-rbd.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout: python-rgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Updated: 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout: librados2.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.765 INFO:teuthology.orchestra.run.smithi199.stdout: librbd1.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.766 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:50.766 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:29:50.821 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-11T16:29:50.822 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:50.822 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:29:50.822 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.822 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:50.822 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:29:50.822 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.822 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.823 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.823 INFO:teuthology.orchestra.run.smithi079.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-12-11T16:29:50.823 INFO:teuthology.orchestra.run.smithi079.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-12-11T16:29:50.823 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.823 INFO:teuthology.orchestra.run.smithi079.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-12-11T16:29:50.823 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-12-11T16:29:50.823 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: python-ceph-argparse.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: python-ceph-common.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: python-cephfs.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-12-11T16:29:50.824 INFO:teuthology.orchestra.run.smithi079.stdout: python-rados.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: python-rbd.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: python-rgw.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-12-11T16:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Updated: 2019-12-11T16:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: librados2.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:29:50.826 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:50.826 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:29:50.896 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-test 2019-12-11T16:29:50.956 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2019-12-11T16:29:51.186 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:29:51.220 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:29:51.230 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:29:51.231 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:29:51.231 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:29:51.231 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:29:51.232 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:29:51.259 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:29:51.260 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:29:51.260 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:29:51.261 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:29:51.261 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:29:51.885 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:29:51.917 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:29:52.228 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:29:52.228 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:29:52.228 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-test.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:52.232 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:29:52.233 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:29:52.233 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-test.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:29:52.246 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:52.251 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:52.275 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:52.276 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:52.278 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:29:52.278 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-12-11T16:29:52.278 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-12-11T16:29:52.279 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-12-11T16:29:52.280 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:52.281 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:29:52.281 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-12-11T16:29:52.282 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:29:52.282 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-12-11T16:29:52.283 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-12-11T16:29:52.283 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-12-11T16:29:52.284 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:29:52.284 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-12-11T16:29:52.285 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-12-11T16:29:52.288 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:29:52.288 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-12-11T16:29:52.643 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:29:52.650 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:29:52.866 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:52.866 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:29:52.867 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:52.867 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:29:52.868 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository 2019-12-11T16:29:52.868 INFO:teuthology.orchestra.run.smithi199.stdout: Size 2019-12-11T16:29:52.868 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:29:52.868 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:29:52.868 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-test x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 38 M 2019-12-11T16:29:52.868 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:29:52.868 INFO:teuthology.orchestra.run.smithi199.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-12-11T16:29:52.869 INFO:teuthology.orchestra.run.smithi199.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-12-11T16:29:52.869 INFO:teuthology.orchestra.run.smithi199.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-12-11T16:29:52.869 INFO:teuthology.orchestra.run.smithi199.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-12-11T16:29:52.869 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:52.869 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:29:52.869 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:29:52.869 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+4 Dependent packages) 2019-12-11T16:29:52.870 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:29:52.875 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 38 M 2019-12-11T16:29:52.876 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 134 M 2019-12-11T16:29:52.876 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:29:52.877 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:52.877 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:29:52.877 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:52.877 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:29:52.877 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout: Size 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 38 M 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-12-11T16:29:52.878 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-12-11T16:29:52.879 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-12-11T16:29:52.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:52.879 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:29:52.879 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:29:52.879 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+4 Dependent packages) 2019-12-11T16:29:52.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:29:52.879 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 38 M 2019-12-11T16:29:52.880 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 134 M 2019-12-11T16:29:52.880 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:29:55.882 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:29:55.882 INFO:teuthology.orchestra.run.smithi199.stdout:Total 13 MB/s | 38 MB 00:03 2019-12-11T16:29:55.885 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:29:55.905 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:29:55.980 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:29:55.980 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:29:55.998 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:29:55.998 INFO:teuthology.orchestra.run.smithi079.stdout:Total 12 MB/s | 38 MB 00:03 2019-12-11T16:29:56.001 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:29:56.021 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:29:56.070 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:29:56.071 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:29:56.671 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2019-12-11T16:29:56.681 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2019-12-11T16:29:56.994 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2019-12-11T16:29:57.053 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2019-12-11T16:29:57.270 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2019-12-11T16:29:57.389 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2019-12-11T16:30:01.243 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : jq-1.5-1.el7.x86_64 4/5 2019-12-11T16:30:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.5-1.el7.x86_64 4/5 2019-12-11T16:30:01.540 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 5/5 2019-12-11T16:30:01.671 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 1/5 2019-12-11T16:30:01.707 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 5/5 2019-12-11T16:30:01.930 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-test-15.0.0-7938.g1b78fd0.el7.x86_64 1/5 2019-12-11T16:30:02.147 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 2/5 2019-12-11T16:30:02.247 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 2/5 2019-12-11T16:30:02.289 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 3/5 2019-12-11T16:30:02.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 3/5 2019-12-11T16:30:02.398 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : jq-1.5-1.el7.x86_64 4/5 2019-12-11T16:30:02.432 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.5-1.el7.x86_64 4/5 2019-12-11T16:30:02.849 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2019-12-11T16:30:02.849 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:02.849 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:30:02.849 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-test.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:02.849 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:02.849 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:30:02.850 INFO:teuthology.orchestra.run.smithi199.stdout: jq.x86_64 0:1.5-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2019-12-11T16:30:02.850 INFO:teuthology.orchestra.run.smithi199.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2019-12-11T16:30:02.850 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:02.851 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2019-12-11T16:30:02.851 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:02.851 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:30:02.851 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:02.851 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:02.851 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:30:02.851 INFO:teuthology.orchestra.run.smithi079.stdout: jq.x86_64 0:1.5-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2019-12-11T16:30:02.852 INFO:teuthology.orchestra.run.smithi079.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2019-12-11T16:30:02.852 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:02.852 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:30:02.852 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:30:02.971 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph 2019-12-11T16:30:02.980 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2019-12-11T16:30:03.213 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:03.253 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:03.273 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:03.319 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:03.320 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:03.322 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:03.322 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:03.323 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:03.907 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:03.968 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:04.230 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:30:04.230 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:04.231 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.233 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.262 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.263 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.264 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.266 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:04.266 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mds.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.275 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.277 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.278 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.280 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.281 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.282 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.283 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.284 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mon.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.286 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-osd.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.291 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:04.291 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-12-11T16:30:04.292 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-12-11T16:30:04.293 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-12-11T16:30:04.293 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.294 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.295 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.296 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.297 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.297 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-requests.noarch 0:2.6.0-7.el7_7 will be installed 2019-12-11T16:30:04.298 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-7.el7_7.noarch 2019-12-11T16:30:04.299 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-12-11T16:30:04.299 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-12-11T16:30:04.300 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-12-11T16:30:04.301 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:04.301 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-12-11T16:30:04.301 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-12-11T16:30:04.301 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-12-11T16:30:04.302 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-12-11T16:30:04.303 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-12-11T16:30:04.303 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-12-11T16:30:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-12-11T16:30:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-12-11T16:30:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-12-11T16:30:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-12-11T16:30:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-12-11T16:30:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-12-11T16:30:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-12-11T16:30:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-12-11T16:30:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-12-11T16:30:04.316 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:30:04.317 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:04.317 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.319 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.350 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.351 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.351 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.353 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:04.353 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mds.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.364 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.365 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.366 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.367 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.368 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.369 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.370 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:30:04.371 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mon.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.374 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-osd.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:04.378 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:04.378 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-12-11T16:30:04.379 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-12-11T16:30:04.380 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-12-11T16:30:04.380 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.381 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.382 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.383 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.384 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-12-11T16:30:04.385 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-requests.noarch 0:2.6.0-7.el7_7 will be installed 2019-12-11T16:30:04.385 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-7.el7_7.noarch 2019-12-11T16:30:04.386 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-12-11T16:30:04.386 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-12-11T16:30:04.387 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-12-11T16:30:04.388 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:04.388 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-12-11T16:30:04.388 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-12-11T16:30:04.389 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-12-11T16:30:04.389 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-12-11T16:30:04.390 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-12-11T16:30:04.391 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-12-11T16:30:04.393 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-12-11T16:30:04.393 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-12-11T16:30:04.393 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-12-11T16:30:04.393 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:04.393 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-12-11T16:30:04.394 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-12-11T16:30:04.394 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-12-11T16:30:04.394 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:04.394 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-12-11T16:30:04.394 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-12-11T16:30:04.394 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:04.394 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-12-11T16:30:04.664 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:04.752 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:04.882 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:04.883 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:30:04.885 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:04.886 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:04.886 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository 2019-12-11T16:30:04.886 INFO:teuthology.orchestra.run.smithi079.stdout: Size 2019-12-11T16:30:04.886 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:04.886 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:30:04.886 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 2.0 k 2019-12-11T16:30:04.886 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:30:04.887 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 1.9 M 2019-12-11T16:30:04.887 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 1.7 M 2019-12-11T16:30:04.887 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 4.4 M 2019-12-11T16:30:04.887 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 16 M 2019-12-11T16:30:04.905 INFO:teuthology.orchestra.run.smithi079.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-12-11T16:30:04.906 INFO:teuthology.orchestra.run.smithi079.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-12-11T16:30:04.906 INFO:teuthology.orchestra.run.smithi079.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-12-11T16:30:04.906 INFO:teuthology.orchestra.run.smithi079.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-12-11T16:30:04.906 INFO:teuthology.orchestra.run.smithi079.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-12-11T16:30:04.906 INFO:teuthology.orchestra.run.smithi079.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-12-11T16:30:04.906 INFO:teuthology.orchestra.run.smithi079.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-requests noarch 2.6.0-7.el7_7 updates 95 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-12-11T16:30:04.907 INFO:teuthology.orchestra.run.smithi079.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-12-11T16:30:04.908 INFO:teuthology.orchestra.run.smithi079.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-12-11T16:30:04.908 INFO:teuthology.orchestra.run.smithi079.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-12-11T16:30:04.908 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:04.908 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:30:04.908 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:04.908 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+21 Dependent packages) 2019-12-11T16:30:04.908 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:04.911 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 28 M 2019-12-11T16:30:04.911 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 87 M 2019-12-11T16:30:04.911 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:30:04.974 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:04.975 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:30:04.977 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:04.977 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:04.978 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository 2019-12-11T16:30:04.978 INFO:teuthology.orchestra.run.smithi199.stdout: Size 2019-12-11T16:30:04.978 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:04.978 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:30:04.978 INFO:teuthology.orchestra.run.smithi199.stdout: ceph x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 2.0 k 2019-12-11T16:30:04.978 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:30:04.978 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mds x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 1.9 M 2019-12-11T16:30:04.979 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 1.7 M 2019-12-11T16:30:04.979 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mon x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 4.4 M 2019-12-11T16:30:04.979 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-osd x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 16 M 2019-12-11T16:30:04.979 INFO:teuthology.orchestra.run.smithi199.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-12-11T16:30:04.979 INFO:teuthology.orchestra.run.smithi199.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-12-11T16:30:04.979 INFO:teuthology.orchestra.run.smithi199.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-12-11T16:30:04.979 INFO:teuthology.orchestra.run.smithi199.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-12-11T16:30:04.980 INFO:teuthology.orchestra.run.smithi199.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-12-11T16:30:04.980 INFO:teuthology.orchestra.run.smithi199.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-12-11T16:30:04.980 INFO:teuthology.orchestra.run.smithi199.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-12-11T16:30:04.980 INFO:teuthology.orchestra.run.smithi199.stdout: python-requests noarch 2.6.0-7.el7_7 updates 95 k 2019-12-11T16:30:04.980 INFO:teuthology.orchestra.run.smithi199.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-12-11T16:30:04.980 INFO:teuthology.orchestra.run.smithi199.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-12-11T16:30:04.980 INFO:teuthology.orchestra.run.smithi199.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-12-11T16:30:04.981 INFO:teuthology.orchestra.run.smithi199.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-12-11T16:30:04.981 INFO:teuthology.orchestra.run.smithi199.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-12-11T16:30:04.981 INFO:teuthology.orchestra.run.smithi199.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-12-11T16:30:04.981 INFO:teuthology.orchestra.run.smithi199.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-12-11T16:30:04.981 INFO:teuthology.orchestra.run.smithi199.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-12-11T16:30:04.981 INFO:teuthology.orchestra.run.smithi199.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-12-11T16:30:04.981 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:04.982 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:30:04.982 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:04.982 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+21 Dependent packages) 2019-12-11T16:30:04.982 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:04.983 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 28 M 2019-12-11T16:30:04.983 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 87 M 2019-12-11T16:30:04.983 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:30:07.184 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:07.185 INFO:teuthology.orchestra.run.smithi079.stdout:Total 12 MB/s | 28 MB 00:02 2019-12-11T16:30:07.202 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:30:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:30:07.344 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:07.344 INFO:teuthology.orchestra.run.smithi199.stdout:Total 12 MB/s | 28 MB 00:02 2019-12-11T16:30:07.371 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:30:07.419 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:30:07.574 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:30:07.574 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:30:07.705 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:30:07.706 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:30:08.502 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-webob-1.2.3-7.el7.noarch 1/22 2019-12-11T16:30:08.610 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-webob-1.2.3-7.el7.noarch 1/22 2019-12-11T16:30:08.848 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 2/22 2019-12-11T16:30:08.932 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 2/22 2019-12-11T16:30:09.320 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/22 2019-12-11T16:30:09.426 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/22 2019-12-11T16:30:09.722 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mds-15.0.0-7938.g1b78fd0.el7.x86_64 4/22 2019-12-11T16:30:09.873 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mds-15.0.0-7938.g1b78fd0.el7.x86_64 4/22 2019-12-11T16:30:10.184 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 5/22 2019-12-11T16:30:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 5/22 2019-12-11T16:30:10.564 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 6/22 2019-12-11T16:30:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 6/22 2019-12-11T16:30:10.975 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 7/22 2019-12-11T16:30:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 7/22 2019-12-11T16:30:11.774 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 8/22 2019-12-11T16:30:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 8/22 2019-12-11T16:30:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mon-15.0.0-7938.g1b78fd0.el7.x86_64 9/22 2019-12-11T16:30:12.563 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mon-15.0.0-7938.g1b78fd0.el7.x86_64 9/22 2019-12-11T16:30:12.831 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 10/22 2019-12-11T16:30:12.848 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 10/22 2019-12-11T16:30:13.189 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 11/22 2019-12-11T16:30:13.232 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 11/22 2019-12-11T16:30:13.476 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-mako-0.8.1-2.el7.noarch 12/22 2019-12-11T16:30:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-mako-0.8.1-2.el7.noarch 12/22 2019-12-11T16:30:15.319 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 13/22 2019-12-11T16:30:15.426 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 13/22 2019-12-11T16:30:15.777 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-osd-15.0.0-7938.g1b78fd0.el7.x86_64 14/22 2019-12-11T16:30:15.886 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-osd-15.0.0-7938.g1b78fd0.el7.x86_64 14/22 2019-12-11T16:30:16.121 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 15/22 2019-12-11T16:30:16.236 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 15/22 2019-12-11T16:30:16.384 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-requests-2.6.0-7.el7_7.noarch 16/22 2019-12-11T16:30:16.508 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-requests-2.6.0-7.el7_7.noarch 16/22 2019-12-11T16:30:16.605 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-six-1.9.0-0.el7.noarch 17/22 2019-12-11T16:30:16.754 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-six-1.9.0-0.el7.noarch 17/22 2019-12-11T16:30:17.027 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 18/22 2019-12-11T16:30:17.145 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 18/22 2019-12-11T16:30:17.539 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 19/22 2019-12-11T16:30:17.628 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 19/22 2019-12-11T16:30:18.247 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 20/22 2019-12-11T16:30:18.363 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 20/22 2019-12-11T16:30:18.610 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 21/22 2019-12-11T16:30:18.724 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 21/22 2019-12-11T16:30:18.881 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 22/22 2019-12-11T16:30:18.990 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 1/22 2019-12-11T16:30:19.003 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 22/22 2019-12-11T16:30:19.116 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 2/22 2019-12-11T16:30:19.156 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 1/22 2019-12-11T16:30:19.233 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 3/22 2019-12-11T16:30:19.268 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 2/22 2019-12-11T16:30:19.333 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 4/22 2019-12-11T16:30:19.383 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 3/22 2019-12-11T16:30:19.426 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 5/22 2019-12-11T16:30:19.491 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 4/22 2019-12-11T16:30:19.551 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 6/22 2019-12-11T16:30:19.591 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 5/22 2019-12-11T16:30:19.685 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 7/22 2019-12-11T16:30:19.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 6/22 2019-12-11T16:30:19.845 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-requests-2.6.0-7.el7_7.noarch 8/22 2019-12-11T16:30:19.859 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 7/22 2019-12-11T16:30:19.968 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-requests-2.6.0-7.el7_7.noarch 8/22 2019-12-11T16:30:19.969 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 9/22 2019-12-11T16:30:20.078 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-osd-15.0.0-7938.g1b78fd0.el7.x86_64 10/22 2019-12-11T16:30:20.093 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 9/22 2019-12-11T16:30:20.202 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-osd-15.0.0-7938.g1b78fd0.el7.x86_64 10/22 2019-12-11T16:30:20.211 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 11/22 2019-12-11T16:30:20.306 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 11/22 2019-12-11T16:30:20.337 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 12/22 2019-12-11T16:30:20.386 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 12/22 2019-12-11T16:30:20.421 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 13/22 2019-12-11T16:30:20.470 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 13/22 2019-12-11T16:30:20.521 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 14/22 2019-12-11T16:30:20.570 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 14/22 2019-12-11T16:30:20.605 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 15/22 2019-12-11T16:30:20.696 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 15/22 2019-12-11T16:30:20.739 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mon-15.0.0-7938.g1b78fd0.el7.x86_64 16/22 2019-12-11T16:30:20.830 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mon-15.0.0-7938.g1b78fd0.el7.x86_64 16/22 2019-12-11T16:30:20.839 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 17/22 2019-12-11T16:30:20.931 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 18/22 2019-12-11T16:30:20.947 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 17/22 2019-12-11T16:30:21.064 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 18/22 2019-12-11T16:30:21.075 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 19/22 2019-12-11T16:30:21.166 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 20/22 2019-12-11T16:30:21.223 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 19/22 2019-12-11T16:30:21.266 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 21/22 2019-12-11T16:30:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-15.0.0-7938.g1b78fd0.el7.x86_64 20/22 2019-12-11T16:30:21.432 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 21/22 2019-12-11T16:30:21.812 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mds-15.0.0-7938.g1b78fd0.el7.x86_64 22/22 2019-12-11T16:30:21.812 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:21.812 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:30:21.812 INFO:teuthology.orchestra.run.smithi199.stdout: ceph.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.812 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:21.812 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:30:21.813 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mds.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.813 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.813 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mon.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.813 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-osd.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.813 INFO:teuthology.orchestra.run.smithi199.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-12-11T16:30:21.813 INFO:teuthology.orchestra.run.smithi199.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-12-11T16:30:21.813 INFO:teuthology.orchestra.run.smithi199.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-12-11T16:30:21.814 INFO:teuthology.orchestra.run.smithi199.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-12-11T16:30:21.814 INFO:teuthology.orchestra.run.smithi199.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-12-11T16:30:21.814 INFO:teuthology.orchestra.run.smithi199.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-12-11T16:30:21.814 INFO:teuthology.orchestra.run.smithi199.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-12-11T16:30:21.814 INFO:teuthology.orchestra.run.smithi199.stdout: python-requests.noarch 0:2.6.0-7.el7_7 2019-12-11T16:30:21.814 INFO:teuthology.orchestra.run.smithi199.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-12-11T16:30:21.814 INFO:teuthology.orchestra.run.smithi199.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-12-11T16:30:21.815 INFO:teuthology.orchestra.run.smithi199.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-12-11T16:30:21.815 INFO:teuthology.orchestra.run.smithi199.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-12-11T16:30:21.815 INFO:teuthology.orchestra.run.smithi199.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-12-11T16:30:21.815 INFO:teuthology.orchestra.run.smithi199.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-12-11T16:30:21.815 INFO:teuthology.orchestra.run.smithi199.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-12-11T16:30:21.815 INFO:teuthology.orchestra.run.smithi199.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-12-11T16:30:21.815 INFO:teuthology.orchestra.run.smithi199.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-12-11T16:30:21.816 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:21.816 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:30:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mds-15.0.0-7938.g1b78fd0.el7.x86_64 22/22 2019-12-11T16:30:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:21.927 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:30:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: ceph.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:21.927 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:30:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-12-11T16:30:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-12-11T16:30:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-12-11T16:30:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-12-11T16:30:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-12-11T16:30:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-12-11T16:30:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-12-11T16:30:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: python-requests.noarch 0:2.6.0-7.el7_7 2019-12-11T16:30:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-12-11T16:30:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-12-11T16:30:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-12-11T16:30:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-12-11T16:30:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-12-11T16:30:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-12-11T16:30:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-12-11T16:30:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-12-11T16:30:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-12-11T16:30:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:21.930 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:30:21.936 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-daemon 2019-12-11T16:30:22.045 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-daemon 2019-12-11T16:30:22.176 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:22.218 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:22.219 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:22.219 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:22.220 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:22.220 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:22.269 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:22.307 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:22.308 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:22.309 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:22.309 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:22.309 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:22.865 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:22.925 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:23.198 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:30:23.198 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:23.198 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:23.249 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:30:23.250 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:23.250 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:23.562 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:23.626 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:23.787 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:23.787 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:30:23.788 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:23.788 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:23.788 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository 2019-12-11T16:30:23.788 INFO:teuthology.orchestra.run.smithi199.stdout: Size 2019-12-11T16:30:23.789 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:23.789 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:30:23.789 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-daemon x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 17 k 2019-12-11T16:30:23.789 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:23.789 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:30:23.789 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:23.789 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package 2019-12-11T16:30:23.790 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:23.790 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 17 k 2019-12-11T16:30:23.790 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 68 k 2019-12-11T16:30:23.790 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:30:23.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:23.845 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:30:23.846 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:23.846 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:23.846 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository 2019-12-11T16:30:23.846 INFO:teuthology.orchestra.run.smithi079.stdout: Size 2019-12-11T16:30:23.846 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:23.846 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:30:23.847 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-daemon x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 17 k 2019-12-11T16:30:23.847 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:23.847 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:30:23.847 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:23.847 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2019-12-11T16:30:23.847 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:23.847 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 17 k 2019-12-11T16:30:23.848 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 68 k 2019-12-11T16:30:23.848 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:30:24.151 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:30:24.159 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:30:24.161 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:30:24.162 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:30:24.208 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:30:24.213 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:30:24.215 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:30:24.215 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:30:24.968 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-daemon-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:30:25.001 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-daemon-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:30:25.433 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-daemon-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:30:25.433 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:25.433 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:30:25.433 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-daemon.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:25.433 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:25.434 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:30:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-daemon-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:30:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:25.489 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:30:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-daemon.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:25.489 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:30:25.561 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-mgr 2019-12-11T16:30:25.609 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2019-12-11T16:30:25.735 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:25.773 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:25.773 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:25.773 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:25.773 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:25.774 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:25.841 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:25.886 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:25.886 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:25.886 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:25.886 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:25.886 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:26.360 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:26.535 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:26.688 INFO:teuthology.orchestra.run.smithi199.stdout:Package 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:30:26.688 INFO:teuthology.orchestra.run.smithi199.stdout:Nothing to do 2019-12-11T16:30:26.751 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-mgr-dashboard 2019-12-11T16:30:26.867 INFO:teuthology.orchestra.run.smithi079.stdout:Package 2:ceph-mgr-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:30:26.867 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do 2019-12-11T16:30:26.927 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2019-12-11T16:30:26.928 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:26.964 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:26.965 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:26.966 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:26.966 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:26.966 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:27.183 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:27.228 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:27.229 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:27.230 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:27.230 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:27.230 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:27.556 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:27.882 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:27.887 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:30:27.888 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:27.888 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:27.891 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:27.924 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:27.924 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:27.924 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:27.924 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-12-11T16:30:27.924 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-12-11T16:30:27.924 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:27.925 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-12-11T16:30:28.221 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:30:28.221 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:28.221 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:28.224 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:28.255 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:28.255 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:28.255 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:28.256 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-12-11T16:30:28.256 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-12-11T16:30:28.257 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:28.257 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-12-11T16:30:28.286 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:28.512 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:28.512 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:30:28.513 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:28.513 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:28.513 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository Size 2019-12-11T16:30:28.513 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:28.513 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:30:28.514 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-dashboard noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 3.9 M 2019-12-11T16:30:28.514 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:30:28.514 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-grafana-dashboards noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 18 k 2019-12-11T16:30:28.514 INFO:teuthology.orchestra.run.smithi199.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-12-11T16:30:28.514 INFO:teuthology.orchestra.run.smithi199.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-12-11T16:30:28.514 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:28.514 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:30:28.515 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:28.515 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+3 Dependent packages) 2019-12-11T16:30:28.515 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:28.515 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 4.6 M 2019-12-11T16:30:28.515 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 20 M 2019-12-11T16:30:28.515 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:30:28.620 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:28.842 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:28.843 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:30:28.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:28.843 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:28.844 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:30:28.844 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:28.844 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:30:28.844 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 3.9 M 2019-12-11T16:30:28.844 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:30:28.844 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 18 k 2019-12-11T16:30:28.844 INFO:teuthology.orchestra.run.smithi079.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-12-11T16:30:28.845 INFO:teuthology.orchestra.run.smithi079.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-12-11T16:30:28.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:28.845 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:30:28.845 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:28.845 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+3 Dependent packages) 2019-12-11T16:30:28.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:28.845 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 4.6 M 2019-12-11T16:30:28.846 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 20 M 2019-12-11T16:30:28.846 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:30:29.801 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:29.802 INFO:teuthology.orchestra.run.smithi199.stdout:Total 3.5 MB/s | 4.6 MB 00:01 2019-12-11T16:30:29.808 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:30:29.817 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:30:29.974 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:30:29.974 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:30:30.073 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:30.073 INFO:teuthology.orchestra.run.smithi079.stdout:Total 3.7 MB/s | 4.6 MB 00:01 2019-12-11T16:30:30.078 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:30:30.087 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:30:30.238 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:30:30.239 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:30:30.643 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-7938.g1b78fd0.el7.noarch 1/4 2019-12-11T16:30:30.900 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-7938.g1b78fd0.el7.noarch 1/4 2019-12-11T16:30:31.081 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 2/4 2019-12-11T16:30:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 2/4 2019-12-11T16:30:32.185 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-routes-1.13-2.el7.noarch 3/4 2019-12-11T16:30:32.462 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-routes-1.13-2.el7.noarch 3/4 2019-12-11T16:30:33.326 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 4/4 2019-12-11T16:30:33.464 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 1/4 2019-12-11T16:30:33.553 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 2/4 2019-12-11T16:30:33.645 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-7938.g1b78fd0.el7.noarch 3/4 2019-12-11T16:30:33.725 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 4/4 2019-12-11T16:30:33.844 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 1/4 2019-12-11T16:30:33.969 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-7938.g1b78fd0.el7.noarch 2/4 2019-12-11T16:30:34.063 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-routes-1.13-2.el7.noarch 4/4 2019-12-11T16:30:34.064 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:34.064 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:30:34.064 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:34.064 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:34.064 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:30:34.064 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:34.065 INFO:teuthology.orchestra.run.smithi199.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-12-11T16:30:34.065 INFO:teuthology.orchestra.run.smithi199.stdout: python-routes.noarch 0:1.13-2.el7 2019-12-11T16:30:34.065 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:34.065 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:30:34.102 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-7938.g1b78fd0.el7.noarch 3/4 2019-12-11T16:30:34.199 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-mgr-diskprediction-cloud 2019-12-11T16:30:34.385 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:34.427 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:34.427 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:34.428 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:34.428 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:34.428 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:34.598 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-routes-1.13-2.el7.noarch 4/4 2019-12-11T16:30:34.600 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:34.600 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:30:34.601 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:34.601 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:34.601 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:30:34.601 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:34.601 INFO:teuthology.orchestra.run.smithi079.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-12-11T16:30:34.601 INFO:teuthology.orchestra.run.smithi079.stdout: python-routes.noarch 0:1.13-2.el7 2019-12-11T16:30:34.601 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:34.602 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:30:34.725 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-cloud 2019-12-11T16:30:34.950 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:34.990 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:35.021 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:35.348 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:30:35.349 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:35.349 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:35.648 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:35.714 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:35.939 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:35.940 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:30:35.940 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:35.940 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:35.941 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository Size 2019-12-11T16:30:35.941 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:35.941 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:30:35.941 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-diskprediction-cloud 2019-12-11T16:30:35.941 INFO:teuthology.orchestra.run.smithi199.stdout: noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 82 k 2019-12-11T16:30:35.941 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:35.941 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:30:35.942 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:35.942 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package 2019-12-11T16:30:35.942 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:35.942 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 82 k 2019-12-11T16:30:35.942 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 411 k 2019-12-11T16:30:35.942 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:30:35.975 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:30:35.975 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:35.975 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:36.240 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:30:36.244 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:30:36.298 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:30:36.299 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:30:36.342 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:36.567 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:36.567 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:30:36.568 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:36.568 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:36.568 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:30:36.569 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:36.569 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:30:36.569 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-cloud 2019-12-11T16:30:36.569 INFO:teuthology.orchestra.run.smithi079.stdout: noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 82 k 2019-12-11T16:30:36.569 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:36.569 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:30:36.569 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:36.570 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2019-12-11T16:30:36.570 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:36.570 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 82 k 2019-12-11T16:30:36.570 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 411 k 2019-12-11T16:30:36.570 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:30:36.907 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:30:36.910 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:30:36.963 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:30:36.964 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:30:36.988 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-7938.g1b78fd0.el7.n 1/1 2019-12-11T16:30:37.491 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-7938.g1b78fd0.el7.n 1/1 2019-12-11T16:30:37.491 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:37.492 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:30:37.492 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:37.492 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:37.492 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:30:37.612 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-mgr-diskprediction-local 2019-12-11T16:30:37.693 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-7938.g1b78fd0.el7.n 1/1 2019-12-11T16:30:37.876 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:37.921 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:37.921 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:37.922 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:37.922 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:37.922 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:38.106 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-7938.g1b78fd0.el7.n 1/1 2019-12-11T16:30:38.106 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:38.107 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:30:38.107 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:38.107 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:38.107 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:30:38.223 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2019-12-11T16:30:38.446 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:38.488 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:38.489 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:38.489 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:38.489 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:38.490 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:38.577 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:38.921 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:30:38.921 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:38.921 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:38.923 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:38.954 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:38.954 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-12-11T16:30:38.960 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-12-11T16:30:38.961 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-11T16:30:38.963 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-11T16:30:38.964 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:38.964 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-12-11T16:30:38.965 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-12-11T16:30:38.966 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-11T16:30:38.967 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-11T16:30:38.968 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:38.968 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-12-11T16:30:39.141 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:39.333 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:39.470 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:30:39.470 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:39.471 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:39.473 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:39.503 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:39.503 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-12-11T16:30:39.510 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-12-11T16:30:39.510 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-11T16:30:39.513 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-11T16:30:39.513 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:39.513 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-12-11T16:30:39.514 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-12-11T16:30:39.515 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-11T16:30:39.517 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-11T16:30:39.517 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:39.517 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-12-11T16:30:39.563 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:39.564 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:30:39.565 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:39.565 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:39.565 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository Size 2019-12-11T16:30:39.566 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:39.566 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:30:39.566 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-diskprediction-local 2019-12-11T16:30:39.567 INFO:teuthology.orchestra.run.smithi199.stdout: noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 8.6 M 2019-12-11T16:30:39.567 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:30:39.567 INFO:teuthology.orchestra.run.smithi199.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-12-11T16:30:39.567 INFO:teuthology.orchestra.run.smithi199.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-12-11T16:30:39.567 INFO:teuthology.orchestra.run.smithi199.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-12-11T16:30:39.567 INFO:teuthology.orchestra.run.smithi199.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-12-11T16:30:39.568 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:39.568 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:30:39.568 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:39.568 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+4 Dependent packages) 2019-12-11T16:30:39.568 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:39.569 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 19 M 2019-12-11T16:30:39.569 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 104 M 2019-12-11T16:30:39.569 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:30:39.879 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:40.100 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:40.101 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:30:40.102 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:40.102 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:40.102 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:30:40.102 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:40.103 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:30:40.103 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local 2019-12-11T16:30:40.103 INFO:teuthology.orchestra.run.smithi079.stdout: noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 8.6 M 2019-12-11T16:30:40.103 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:30:40.103 INFO:teuthology.orchestra.run.smithi079.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-12-11T16:30:40.103 INFO:teuthology.orchestra.run.smithi079.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-12-11T16:30:40.104 INFO:teuthology.orchestra.run.smithi079.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-12-11T16:30:40.104 INFO:teuthology.orchestra.run.smithi079.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-12-11T16:30:40.104 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:40.104 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:30:40.104 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:40.104 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+4 Dependent packages) 2019-12-11T16:30:40.105 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:40.105 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 19 M 2019-12-11T16:30:40.105 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 104 M 2019-12-11T16:30:40.105 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:30:40.528 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:40.529 INFO:teuthology.orchestra.run.smithi199.stdout:Total 20 MB/s | 19 MB 00:00 2019-12-11T16:30:40.539 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:30:40.563 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:30:40.699 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:30:40.699 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:30:41.082 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:41.082 INFO:teuthology.orchestra.run.smithi079.stdout:Total 20 MB/s | 19 MB 00:00 2019-12-11T16:30:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:30:41.109 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:30:41.208 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:30:41.209 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:30:41.556 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2019-12-11T16:30:42.005 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2019-12-11T16:30:42.021 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2019-12-11T16:30:42.489 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2019-12-11T16:30:44.371 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-11T16:30:44.484 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-11T16:30:46.032 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-11T16:30:46.170 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-11T16:30:46.367 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-7938.g1b78fd0.el7.n 5/5 2019-12-11T16:30:46.492 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-7938.g1b78fd0.el7.n 5/5 2019-12-11T16:30:46.497 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 1/5 2019-12-11T16:30:46.621 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 1/5 2019-12-11T16:30:46.622 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-7938.g1b78fd0.el7.n 2/5 2019-12-11T16:30:46.713 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-7938.g1b78fd0.el7.n 2/5 2019-12-11T16:30:46.773 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-11T16:30:46.805 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-11T16:30:46.873 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-11T16:30:46.897 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-11T16:30:47.349 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2019-12-11T16:30:47.349 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:47.349 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:30:47.350 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:47.350 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:47.350 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:30:47.350 INFO:teuthology.orchestra.run.smithi199.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2019-12-11T16:30:47.350 INFO:teuthology.orchestra.run.smithi199.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2019-12-11T16:30:47.350 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:47.350 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:30:47.374 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2019-12-11T16:30:47.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:47.375 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:30:47.375 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:47.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:47.375 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:30:47.375 INFO:teuthology.orchestra.run.smithi079.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2019-12-11T16:30:47.376 INFO:teuthology.orchestra.run.smithi079.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2019-12-11T16:30:47.376 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:47.376 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:30:47.480 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-mgr-rook 2019-12-11T16:30:47.505 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2019-12-11T16:30:47.661 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:47.700 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:47.700 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:47.701 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:47.701 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:47.701 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:47.729 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:47.770 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:47.771 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:47.772 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:47.772 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:47.775 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:48.311 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:48.419 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:48.642 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:30:48.642 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:48.642 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:48.642 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:48.672 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:48.672 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-1.el7 will be installed 2019-12-11T16:30:48.674 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.675 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.676 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.677 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.678 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:48.678 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-12-11T16:30:48.679 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-12-11T16:30:48.679 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-12-11T16:30:48.680 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-11T16:30:48.680 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-11T16:30:48.681 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-11T16:30:48.682 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-12-11T16:30:48.682 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-12-11T16:30:48.683 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:48.684 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-12-11T16:30:48.684 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-12-11T16:30:48.684 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-12-11T16:30:48.685 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-12-11T16:30:48.685 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-12-11T16:30:48.686 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:48.686 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-12-11T16:30:48.745 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:30:48.746 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:48.746 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:48.747 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:48.781 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:48.785 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-1.el7 will be installed 2019-12-11T16:30:48.785 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.786 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.787 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.787 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-11T16:30:48.787 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:48.787 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-12-11T16:30:48.787 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-12-11T16:30:48.787 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-12-11T16:30:48.787 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-11T16:30:48.788 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-11T16:30:48.788 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-11T16:30:48.789 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-12-11T16:30:48.789 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-12-11T16:30:48.797 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:48.797 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-12-11T16:30:48.797 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-12-11T16:30:48.797 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-12-11T16:30:48.797 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-12-11T16:30:48.797 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-12-11T16:30:48.798 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:48.798 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-12-11T16:30:49.050 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:49.153 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:49.276 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:49.277 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:30:49.278 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:49.278 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:49.279 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository Size 2019-12-11T16:30:49.279 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:49.279 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:30:49.279 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-rook noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 23 k 2019-12-11T16:30:49.279 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:30:49.280 INFO:teuthology.orchestra.run.smithi199.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-12-11T16:30:49.280 INFO:teuthology.orchestra.run.smithi199.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-12-11T16:30:49.280 INFO:teuthology.orchestra.run.smithi199.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-12-11T16:30:49.280 INFO:teuthology.orchestra.run.smithi199.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-12-11T16:30:49.281 INFO:teuthology.orchestra.run.smithi199.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-12-11T16:30:49.281 INFO:teuthology.orchestra.run.smithi199.stdout: python2-kubernetes noarch 10.0.1-1.el7 epel 1.1 M 2019-12-11T16:30:49.281 INFO:teuthology.orchestra.run.smithi199.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-12-11T16:30:49.282 INFO:teuthology.orchestra.run.smithi199.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-12-11T16:30:49.282 INFO:teuthology.orchestra.run.smithi199.stdout: python2-requests-oauthlib noarch 0.8.0-5.el7 base 35 k 2019-12-11T16:30:49.282 INFO:teuthology.orchestra.run.smithi199.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-12-11T16:30:49.282 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:49.283 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:30:49.283 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:49.283 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+10 Dependent packages) 2019-12-11T16:30:49.284 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:49.285 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 1.6 M 2019-12-11T16:30:49.285 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 24 M 2019-12-11T16:30:49.285 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:30:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:49.376 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:30:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:49.378 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:30:49.378 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:49.378 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:30:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 23 k 2019-12-11T16:30:49.378 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:30:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-12-11T16:30:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-12-11T16:30:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-12-11T16:30:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-12-11T16:30:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-12-11T16:30:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: python2-kubernetes noarch 10.0.1-1.el7 epel 1.1 M 2019-12-11T16:30:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-12-11T16:30:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-12-11T16:30:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: python2-requests-oauthlib noarch 0.8.0-5.el7 base 35 k 2019-12-11T16:30:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-12-11T16:30:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:49.380 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:30:49.381 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:49.381 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+10 Dependent packages) 2019-12-11T16:30:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:49.382 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 1.6 M 2019-12-11T16:30:49.382 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 24 M 2019-12-11T16:30:49.382 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:30:49.736 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:49.736 INFO:teuthology.orchestra.run.smithi199.stdout:Total 3.6 MB/s | 1.6 MB 00:00 2019-12-11T16:30:49.745 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:30:49.751 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:30:49.804 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:30:49.804 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:30:49.841 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:49.842 INFO:teuthology.orchestra.run.smithi079.stdout:Total 3.6 MB/s | 1.6 MB 00:00 2019-12-11T16:30:49.853 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:30:49.862 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:30:49.923 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:30:49.923 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:30:50.458 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 1/11 2019-12-11T16:30:50.689 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 1/11 2019-12-11T16:30:50.889 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 2/11 2019-12-11T16:30:51.089 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 2/11 2019-12-11T16:30:51.226 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 3/11 2019-12-11T16:30:51.416 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 3/11 2019-12-11T16:30:51.508 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 4/11 2019-12-11T16:30:51.739 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 4/11 2019-12-11T16:30:51.833 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2019-12-11T16:30:52.079 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2019-12-11T16:30:52.145 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-12-11T16:30:52.449 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-12-11T16:30:52.487 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2019-12-11T16:30:52.961 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2019-12-11T16:30:52.980 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2019-12-11T16:30:53.325 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2019-12-11T16:30:54.272 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2019-12-11T16:30:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2019-12-11T16:30:54.641 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python2-kubernetes-10.0.1-1.el7.noarch 10/11 2019-12-11T16:30:54.885 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mgr-rook-15.0.0-7938.g1b78fd0.el7.noarch 11/11 2019-12-11T16:30:55.001 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2019-12-11T16:30:55.019 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python2-kubernetes-10.0.1-1.el7.noarch 10/11 2019-12-11T16:30:55.152 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 2/11 2019-12-11T16:30:55.269 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 3/11 2019-12-11T16:30:55.307 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mgr-rook-15.0.0-7938.g1b78fd0.el7.noarch 11/11 2019-12-11T16:30:55.361 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 4/11 2019-12-11T16:30:55.413 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2019-12-11T16:30:55.511 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 5/11 2019-12-11T16:30:55.522 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 2/11 2019-12-11T16:30:55.645 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 6/11 2019-12-11T16:30:55.657 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 3/11 2019-12-11T16:30:55.764 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 4/11 2019-12-11T16:30:55.770 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 7/11 2019-12-11T16:30:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 5/11 2019-12-11T16:30:55.904 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python2-kubernetes-10.0.1-1.el7.noarch 8/11 2019-12-11T16:30:56.005 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 9/11 2019-12-11T16:30:56.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 6/11 2019-12-11T16:30:56.116 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 7/11 2019-12-11T16:30:56.122 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 10/11 2019-12-11T16:30:56.258 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python2-kubernetes-10.0.1-1.el7.noarch 8/11 2019-12-11T16:30:56.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 9/11 2019-12-11T16:30:56.459 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 10/11 2019-12-11T16:30:56.599 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-7938.g1b78fd0.el7.noarch 11/11 2019-12-11T16:30:56.599 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:56.599 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:30:56.599 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-rook.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:56.600 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:56.600 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:30:56.600 INFO:teuthology.orchestra.run.smithi199.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-12-11T16:30:56.600 INFO:teuthology.orchestra.run.smithi199.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-12-11T16:30:56.600 INFO:teuthology.orchestra.run.smithi199.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-12-11T16:30:56.600 INFO:teuthology.orchestra.run.smithi199.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-12-11T16:30:56.600 INFO:teuthology.orchestra.run.smithi199.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-12-11T16:30:56.601 INFO:teuthology.orchestra.run.smithi199.stdout: python2-kubernetes.noarch 0:10.0.1-1.el7 2019-12-11T16:30:56.601 INFO:teuthology.orchestra.run.smithi199.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-12-11T16:30:56.601 INFO:teuthology.orchestra.run.smithi199.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-12-11T16:30:56.601 INFO:teuthology.orchestra.run.smithi199.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-12-11T16:30:56.601 INFO:teuthology.orchestra.run.smithi199.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-12-11T16:30:56.601 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:56.601 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:30:56.724 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-mgr-ssh 2019-12-11T16:30:56.912 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-7938.g1b78fd0.el7.noarch 11/11 2019-12-11T16:30:56.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:56.912 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:30:56.912 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:30:56.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:56.912 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:30:56.913 INFO:teuthology.orchestra.run.smithi079.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-12-11T16:30:56.913 INFO:teuthology.orchestra.run.smithi079.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-12-11T16:30:56.913 INFO:teuthology.orchestra.run.smithi079.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-12-11T16:30:56.913 INFO:teuthology.orchestra.run.smithi079.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-12-11T16:30:56.913 INFO:teuthology.orchestra.run.smithi079.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-12-11T16:30:56.913 INFO:teuthology.orchestra.run.smithi079.stdout: python2-kubernetes.noarch 0:10.0.1-1.el7 2019-12-11T16:30:56.913 INFO:teuthology.orchestra.run.smithi079.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-12-11T16:30:56.914 INFO:teuthology.orchestra.run.smithi079.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-12-11T16:30:56.914 INFO:teuthology.orchestra.run.smithi079.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-12-11T16:30:56.914 INFO:teuthology.orchestra.run.smithi079.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-12-11T16:30:56.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:56.914 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:30:56.982 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:57.026 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:57.027 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:30:57.027 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:57.028 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:57.028 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:57.041 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-ssh 2019-12-11T16:30:57.285 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:30:57.327 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:30:57.328 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:30:57.328 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:30:57.329 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:30:57.329 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:30:57.677 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:57.976 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:30:58.014 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:30:58.015 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:58.015 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:58.017 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:58.048 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:58.049 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-12-11T16:30:58.050 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-12-11T16:30:58.050 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:30:58.050 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-12-11T16:30:58.311 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:30:58.312 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:58.312 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:30:58.314 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-7938.g1b78fd0.el7.noarch 2019-12-11T16:30:58.344 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:58.344 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-12-11T16:30:58.345 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-12-11T16:30:58.346 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:30:58.346 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-12-11T16:30:58.417 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:58.642 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:58.643 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:30:58.643 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:58.643 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:58.643 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository Size 2019-12-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-ssh noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 33 k 2019-12-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi199.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-12-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi199.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-12-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:58.645 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:30:58.645 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:30:58.645 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+2 Dependent packages) 2019-12-11T16:30:58.645 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:30:58.646 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 316 k 2019-12-11T16:30:58.646 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 1.1 M 2019-12-11T16:30:58.646 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:30:58.710 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:30:58.907 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:58.908 INFO:teuthology.orchestra.run.smithi199.stdout:Total 1.2 MB/s | 316 kB 00:00 2019-12-11T16:30:58.911 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:30:58.920 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:30:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:58.934 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:30:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:58.935 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:30:58.935 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:58.936 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:30:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-ssh noarch 2:15.0.0-7938.g1b78fd0.el7 ceph-noarch 33 k 2019-12-11T16:30:58.936 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:30:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-12-11T16:30:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-12-11T16:30:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:58.936 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:30:58.937 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:30:58.937 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+2 Dependent packages) 2019-12-11T16:30:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:30:58.937 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 316 k 2019-12-11T16:30:58.937 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.1 M 2019-12-11T16:30:58.937 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:30:58.996 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:30:58.997 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:30:59.207 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:30:59.207 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.2 MB/s | 316 kB 00:00 2019-12-11T16:30:59.208 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:30:59.208 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:30:59.277 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:30:59.278 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:30:59.765 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 1/3 2019-12-11T16:31:00.023 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 2/3 2019-12-11T16:31:00.072 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 1/3 2019-12-11T16:31:00.308 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-7938.g1b78fd0.el7.noarch 3/3 2019-12-11T16:31:00.377 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 2/3 2019-12-11T16:31:00.437 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 1/3 2019-12-11T16:31:00.570 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 2/3 2019-12-11T16:31:00.654 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-7938.g1b78fd0.el7.noarch 3/3 2019-12-11T16:31:00.800 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 1/3 2019-12-11T16:31:00.926 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 2/3 2019-12-11T16:31:01.056 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-7938.g1b78fd0.el7.noarch 3/3 2019-12-11T16:31:01.056 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:01.056 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:31:01.056 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-mgr-ssh.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:01.057 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:01.057 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:31:01.057 INFO:teuthology.orchestra.run.smithi199.stdout: python-execnet.noarch 0:1.2.0-7.el7 python-remoto.noarch 0:0.0.25-1.el7 2019-12-11T16:31:01.057 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:01.057 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:31:01.180 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install ceph-fuse 2019-12-11T16:31:01.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-7938.g1b78fd0.el7.noarch 3/3 2019-12-11T16:31:01.361 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:01.361 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:31:01.361 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-ssh.noarch 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:01.362 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:01.362 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:31:01.362 INFO:teuthology.orchestra.run.smithi079.stdout: python-execnet.noarch 0:1.2.0-7.el7 python-remoto.noarch 0:0.0.25-1.el7 2019-12-11T16:31:01.362 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:01.362 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:31:01.445 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:01.490 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:01.491 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2019-12-11T16:31:01.492 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:31:01.492 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:01.492 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:01.492 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:01.720 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:01.759 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:31:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:02.148 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:02.406 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:02.479 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:31:02.480 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:31:02.480 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:02.490 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python36 for package: 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:31:02.523 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:31:02.523 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:31:02.523 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-12-11T16:31:02.524 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.526 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.527 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.528 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.528 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:31:02.528 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-12-11T16:31:02.534 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-12-11T16:31:02.534 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-12-11T16:31:02.731 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:31:02.731 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:31:02.732 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:02.742 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python36 for package: 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:31:02.774 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:31:02.775 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:31:02.776 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-12-11T16:31:02.776 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.778 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.779 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.780 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64 2019-12-11T16:31:02.780 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:31:02.780 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-12-11T16:31:02.786 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-12-11T16:31:02.786 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-12-11T16:31:02.899 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:03.126 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:03.126 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:31:03.127 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:03.127 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:03.127 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository 2019-12-11T16:31:03.127 INFO:teuthology.orchestra.run.smithi199.stdout: Size 2019-12-11T16:31:03.128 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:03.128 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:31:03.128 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-fuse x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 566 k 2019-12-11T16:31:03.128 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:31:03.128 INFO:teuthology.orchestra.run.smithi199.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-12-11T16:31:03.128 INFO:teuthology.orchestra.run.smithi199.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-12-11T16:31:03.128 INFO:teuthology.orchestra.run.smithi199.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+4 Dependent packages) 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 10 M 2019-12-11T16:31:03.129 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 49 M 2019-12-11T16:31:03.130 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:31:03.147 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:03.370 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:03.370 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:31:03.373 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:03.373 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:03.373 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository 2019-12-11T16:31:03.373 INFO:teuthology.orchestra.run.smithi079.stdout: Size 2019-12-11T16:31:03.374 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:03.374 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:31:03.374 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 566 k 2019-12-11T16:31:03.374 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:31:03.374 INFO:teuthology.orchestra.run.smithi079.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-12-11T16:31:03.374 INFO:teuthology.orchestra.run.smithi079.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-12-11T16:31:03.374 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+4 Dependent packages) 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 10 M 2019-12-11T16:31:03.375 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 49 M 2019-12-11T16:31:03.376 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:31:07.555 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:31:07.555 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.4 MB/s | 10 MB 00:04 2019-12-11T16:31:07.558 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:31:07.558 INFO:teuthology.orchestra.run.smithi199.stdout:Total 2.2 MB/s | 10 MB 00:04 2019-12-11T16:31:07.579 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:31:07.579 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:31:07.608 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:31:07.610 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:31:07.750 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:31:07.750 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:31:07.760 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:31:07.760 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:31:10.039 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 1/5 2019-12-11T16:31:10.112 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 1/5 2019-12-11T16:31:10.527 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python3-3.6.8-10.el7.x86_64 2/5 2019-12-11T16:31:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-3.6.8-10.el7.x86_64 2/5 2019-12-11T16:31:11.421 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 3/5 2019-12-11T16:31:11.526 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 3/5 2019-12-11T16:31:11.885 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 4/5 2019-12-11T16:31:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 4/5 2019-12-11T16:31:12.132 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 5/5 2019-12-11T16:31:12.223 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 1/5 2019-12-11T16:31:12.258 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 5/5 2019-12-11T16:31:12.332 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 2/5 2019-12-11T16:31:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:ceph-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 1/5 2019-12-11T16:31:12.457 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 3/5 2019-12-11T16:31:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 2/5 2019-12-11T16:31:12.615 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 3/5 2019-12-11T16:31:12.616 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python3-3.6.8-10.el7.x86_64 4/5 2019-12-11T16:31:12.850 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-3.6.8-10.el7.x86_64 4/5 2019-12-11T16:31:13.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 5/5 2019-12-11T16:31:13.360 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:13.360 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:31:13.360 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:13.360 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:13.361 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:31:13.361 INFO:teuthology.orchestra.run.smithi079.stdout: python3.x86_64 0:3.6.8-10.el7 python3-libs.x86_64 0:3.6.8-10.el7 2019-12-11T16:31:13.361 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pip.noarch 0:9.0.3-5.el7 python3-setuptools.noarch 0:39.2.0-10.el7 2019-12-11T16:31:13.361 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:13.361 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:31:13.417 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 5/5 2019-12-11T16:31:13.418 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:13.418 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:31:13.418 INFO:teuthology.orchestra.run.smithi199.stdout: ceph-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:13.418 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:13.418 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:31:13.418 INFO:teuthology.orchestra.run.smithi199.stdout: python3.x86_64 0:3.6.8-10.el7 python3-libs.x86_64 0:3.6.8-10.el7 2019-12-11T16:31:13.419 INFO:teuthology.orchestra.run.smithi199.stdout: python3-pip.noarch 0:9.0.3-5.el7 python3-setuptools.noarch 0:39.2.0-10.el7 2019-12-11T16:31:13.419 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:13.419 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:31:13.495 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2019-12-11T16:31:13.554 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install libcephfs2 2019-12-11T16:31:13.736 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:13.767 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:13.777 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:13.778 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:31:13.779 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:13.779 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:13.779 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:13.805 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:13.805 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:31:13.806 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:13.806 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:13.806 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:14.424 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:14.429 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:14.750 INFO:teuthology.orchestra.run.smithi079.stdout:Package 2:libcephfs2-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:31:14.750 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do 2019-12-11T16:31:14.773 INFO:teuthology.orchestra.run.smithi199.stdout:Package 2:libcephfs2-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:31:14.774 INFO:teuthology.orchestra.run.smithi199.stdout:Nothing to do 2019-12-11T16:31:14.811 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2019-12-11T16:31:14.819 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install libcephfs-devel 2019-12-11T16:31:15.017 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:15.046 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:15.047 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:31:15.048 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:15.048 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:15.048 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:15.088 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:15.089 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:31:15.089 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:15.090 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:15.090 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:15.633 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:15.738 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:15.961 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:31:15.962 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:31:15.962 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:15.963 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:libcephfs-devel-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:31:16.000 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:31:16.001 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package librados-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:16.072 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:31:16.073 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:31:16.073 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:16.073 INFO:teuthology.orchestra.run.smithi199.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-7938.g1b78fd0.el7 for package: 2:libcephfs-devel-15.0.0-7938.g1b78fd0.el7.x86_64 2019-12-11T16:31:16.104 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:31:16.104 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package librados-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:16.360 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:16.480 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:16.584 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:16.584 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:31:16.585 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:16.585 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:16.585 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository 2019-12-11T16:31:16.586 INFO:teuthology.orchestra.run.smithi079.stdout: Size 2019-12-11T16:31:16.586 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:16.586 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:31:16.586 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 17 k 2019-12-11T16:31:16.586 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:31:16.586 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 153 k 2019-12-11T16:31:16.587 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:16.587 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:31:16.587 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:16.587 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package (+1 Dependent package) 2019-12-11T16:31:16.587 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:16.587 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 170 k 2019-12-11T16:31:16.588 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 679 k 2019-12-11T16:31:16.588 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:31:16.702 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:16.702 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:31:16.703 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:16.703 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:16.704 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository 2019-12-11T16:31:16.704 INFO:teuthology.orchestra.run.smithi199.stdout: Size 2019-12-11T16:31:16.704 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:16.704 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:31:16.705 INFO:teuthology.orchestra.run.smithi199.stdout: libcephfs-devel x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 17 k 2019-12-11T16:31:16.705 INFO:teuthology.orchestra.run.smithi199.stdout:Installing for dependencies: 2019-12-11T16:31:16.705 INFO:teuthology.orchestra.run.smithi199.stdout: librados-devel x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 153 k 2019-12-11T16:31:16.705 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:16.705 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:31:16.706 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:16.706 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package (+1 Dependent package) 2019-12-11T16:31:16.706 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:16.706 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 170 k 2019-12-11T16:31:16.706 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 679 k 2019-12-11T16:31:16.707 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:31:16.955 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:31:16.955 INFO:teuthology.orchestra.run.smithi079.stdout:Total 460 kB/s | 170 kB 00:00 2019-12-11T16:31:16.958 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:31:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:31:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:31:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:31:17.052 INFO:teuthology.orchestra.run.smithi199.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:31:17.052 INFO:teuthology.orchestra.run.smithi199.stdout:Total 488 kB/s | 170 kB 00:00 2019-12-11T16:31:17.054 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:31:17.065 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:31:17.068 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:31:17.068 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:31:17.585 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:librados-devel-15.0.0-7938.g1b78fd0.el7.x86_64 1/2 2019-12-11T16:31:17.670 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:librados-devel-15.0.0-7938.g1b78fd0.el7.x86_64 1/2 2019-12-11T16:31:18.586 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:libcephfs-devel-15.0.0-7938.g1b78fd0.el7.x86_64 2/2 2019-12-11T16:31:18.688 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:libcephfs-devel-15.0.0-7938.g1b78fd0.el7.x86_64 2/2 2019-12-11T16:31:18.717 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:libcephfs-devel-15.0.0-7938.g1b78fd0.el7.x86_64 1/2 2019-12-11T16:31:18.799 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:libcephfs-devel-15.0.0-7938.g1b78fd0.el7.x86_64 1/2 2019-12-11T16:31:19.235 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:librados-devel-15.0.0-7938.g1b78fd0.el7.x86_64 2/2 2019-12-11T16:31:19.235 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:19.235 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:31:19.236 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:19.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:19.236 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:31:19.236 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:19.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:19.236 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:31:19.283 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:librados-devel-15.0.0-7938.g1b78fd0.el7.x86_64 2/2 2019-12-11T16:31:19.283 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:19.283 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:31:19.283 INFO:teuthology.orchestra.run.smithi199.stdout: libcephfs-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:19.283 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:19.283 INFO:teuthology.orchestra.run.smithi199.stdout:Dependency Installed: 2019-12-11T16:31:19.284 INFO:teuthology.orchestra.run.smithi199.stdout: librados-devel.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:19.284 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:19.284 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:31:19.354 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2019-12-11T16:31:19.404 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install librados2 2019-12-11T16:31:19.578 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:19.621 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:19.622 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:31:19.622 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:19.623 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:19.623 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:19.671 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:19.727 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:19.727 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:31:19.727 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:19.727 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:19.727 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:20.269 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:20.373 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:20.601 INFO:teuthology.orchestra.run.smithi079.stdout:Package 2:librados2-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:31:20.601 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do 2019-12-11T16:31:20.661 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2019-12-11T16:31:20.713 INFO:teuthology.orchestra.run.smithi199.stdout:Package 2:librados2-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:31:20.713 INFO:teuthology.orchestra.run.smithi199.stdout:Nothing to do 2019-12-11T16:31:20.776 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install librbd1 2019-12-11T16:31:20.843 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:20.878 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:20.879 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:31:20.879 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:20.880 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:20.880 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:20.937 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:20.974 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:20.975 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:31:20.975 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:20.975 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:20.975 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:21.473 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:21.565 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:21.791 INFO:teuthology.orchestra.run.smithi079.stdout:Package 2:librbd1-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:31:21.791 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do 2019-12-11T16:31:21.853 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install python-ceph 2019-12-11T16:31:21.893 INFO:teuthology.orchestra.run.smithi199.stdout:Package 2:librbd1-15.0.0-7938.g1b78fd0.el7.x86_64 already installed and latest version 2019-12-11T16:31:21.894 INFO:teuthology.orchestra.run.smithi199.stdout:Nothing to do 2019-12-11T16:31:21.956 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install python-ceph 2019-12-11T16:31:22.060 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:22.099 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:22.099 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:31:22.099 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:22.099 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:22.099 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:22.122 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:22.157 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:22.157 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:31:22.158 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:22.158 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:22.158 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:22.682 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:22.746 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:23.006 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:31:23.007 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:31:23.007 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:23.075 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:31:23.075 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:31:23.076 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:23.375 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:23.443 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:23.558 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:23.558 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:31:23.559 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:23.559 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:23.559 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository 2019-12-11T16:31:23.559 INFO:teuthology.orchestra.run.smithi079.stdout: Size 2019-12-11T16:31:23.560 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:23.560 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:31:23.560 INFO:teuthology.orchestra.run.smithi079.stdout: python-ceph-compat x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 2.2 k 2019-12-11T16:31:23.560 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:23.560 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:31:23.560 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:23.560 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2019-12-11T16:31:23.561 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:23.561 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.2 k 2019-12-11T16:31:23.561 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2019-12-11T16:31:23.561 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:31:23.629 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:23.629 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:31:23.629 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:23.630 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:23.630 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository 2019-12-11T16:31:23.630 INFO:teuthology.orchestra.run.smithi199.stdout: Size 2019-12-11T16:31:23.630 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:23.630 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:31:23.630 INFO:teuthology.orchestra.run.smithi199.stdout: python-ceph-compat x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 2.2 k 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 2.2 k 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 0 2019-12-11T16:31:23.631 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:31:23.859 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:31:23.861 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:31:23.862 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:31:23.863 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:31:23.902 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:31:23.904 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:31:23.905 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:31:23.906 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:31:24.478 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:python-ceph-compat-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:24.493 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:python-ceph-compat-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:24.903 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:python-ceph-compat-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:24.903 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:24.903 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:31:24.903 INFO:teuthology.orchestra.run.smithi199.stdout: python-ceph-compat.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:24.903 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:24.904 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:31:24.904 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:python-ceph-compat-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:24.904 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:24.904 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:31:24.904 INFO:teuthology.orchestra.run.smithi079.stdout: python-ceph-compat.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:24.905 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:24.905 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:31:25.013 INFO:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2019-12-11T16:31:25.014 INFO:teuthology.orchestra.run.smithi199:> sudo yum -y install rbd-fuse 2019-12-11T16:31:25.229 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:25.268 INFO:teuthology.orchestra.run.smithi199.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:31:25.271 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:25.272 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:31:25.273 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:25.273 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:25.273 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:25.312 INFO:teuthology.orchestra.run.smithi199.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:31:25.313 INFO:teuthology.orchestra.run.smithi199.stdout: * base: mirror.facebook.net 2019-12-11T16:31:25.313 INFO:teuthology.orchestra.run.smithi199.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:31:25.313 INFO:teuthology.orchestra.run.smithi199.stdout: * extras: mirror.facebook.net 2019-12-11T16:31:25.314 INFO:teuthology.orchestra.run.smithi199.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:31:25.920 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:25.965 INFO:teuthology.orchestra.run.smithi199.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:31:26.254 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:31:26.255 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:31:26.255 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:26.300 INFO:teuthology.orchestra.run.smithi199.stdout:Resolving Dependencies 2019-12-11T16:31:26.300 INFO:teuthology.orchestra.run.smithi199.stdout:--> Running transaction check 2019-12-11T16:31:26.300 INFO:teuthology.orchestra.run.smithi199.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 will be installed 2019-12-11T16:31:26.628 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:26.680 INFO:teuthology.orchestra.run.smithi199.stdout:--> Finished Dependency Resolution 2019-12-11T16:31:26.813 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:26.813 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:31:26.813 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:26.814 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:26.814 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:31:26.814 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:26.814 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:31:26.814 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 76 k 2019-12-11T16:31:26.814 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:26.815 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:31:26.815 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:31:26.815 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2019-12-11T16:31:26.815 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:26.815 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 76 k 2019-12-11T16:31:26.815 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 200 k 2019-12-11T16:31:26.815 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:31:26.862 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:26.862 INFO:teuthology.orchestra.run.smithi199.stdout:Dependencies Resolved 2019-12-11T16:31:26.862 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:26.862 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:26.862 INFO:teuthology.orchestra.run.smithi199.stdout: Package Arch Version Repository Size 2019-12-11T16:31:26.863 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:26.863 INFO:teuthology.orchestra.run.smithi199.stdout:Installing: 2019-12-11T16:31:26.863 INFO:teuthology.orchestra.run.smithi199.stdout: rbd-fuse x86_64 2:15.0.0-7938.g1b78fd0.el7 ceph 76 k 2019-12-11T16:31:26.863 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:26.863 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction Summary 2019-12-11T16:31:26.863 INFO:teuthology.orchestra.run.smithi199.stdout:================================================================================ 2019-12-11T16:31:26.863 INFO:teuthology.orchestra.run.smithi199.stdout:Install 1 Package 2019-12-11T16:31:26.864 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:26.864 INFO:teuthology.orchestra.run.smithi199.stdout:Total download size: 76 k 2019-12-11T16:31:26.864 INFO:teuthology.orchestra.run.smithi199.stdout:Installed size: 200 k 2019-12-11T16:31:26.864 INFO:teuthology.orchestra.run.smithi199.stdout:Downloading packages: 2019-12-11T16:31:27.145 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:31:27.164 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:31:27.165 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:31:27.166 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:31:27.246 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction check 2019-12-11T16:31:27.268 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction test 2019-12-11T16:31:27.268 INFO:teuthology.orchestra.run.smithi199.stdout:Transaction test succeeded 2019-12-11T16:31:27.269 INFO:teuthology.orchestra.run.smithi199.stdout:Running transaction 2019-12-11T16:31:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : 2:rbd-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:27.839 INFO:teuthology.orchestra.run.smithi199.stdout: Installing : 2:rbd-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:28.303 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : 2:rbd-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:28.303 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:28.303 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:31:28.303 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:28.303 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:31:28.304 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:31:28.416 DEBUG:teuthology.parallel:result is None 2019-12-11T16:31:28.534 INFO:teuthology.orchestra.run.smithi199.stdout: Verifying : 2:rbd-fuse-15.0.0-7938.g1b78fd0.el7.x86_64 1/1 2019-12-11T16:31:28.534 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:28.534 INFO:teuthology.orchestra.run.smithi199.stdout:Installed: 2019-12-11T16:31:28.534 INFO:teuthology.orchestra.run.smithi199.stdout: rbd-fuse.x86_64 2:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:28.534 INFO:teuthology.orchestra.run.smithi199.stdout: 2019-12-11T16:31:28.535 INFO:teuthology.orchestra.run.smithi199.stdout:Complete! 2019-12-11T16:31:28.642 DEBUG:teuthology.parallel:result is None 2019-12-11T16:31:28.643 INFO:teuthology.orchestra.run.smithi079:> sudo lsb_release -is 2019-12-11T16:31:28.795 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS 2019-12-11T16:31:28.797 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-11T16:31:28.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=1b78fd04fb1980223b57f0a3f644592e4e26b518 2019-12-11T16:31:28.951 INFO:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-12-11T16:31:29.236 INFO:teuthology.orchestra.run.smithi079.stdout:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:29.237 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:29.237 INFO:teuthology.task.install:The correct ceph version 15.0.0-7938.g1b78fd0 is installed. 2019-12-11T16:31:29.237 INFO:teuthology.orchestra.run.smithi199:> sudo lsb_release -is 2019-12-11T16:31:29.389 INFO:teuthology.orchestra.run.smithi199.stdout:CentOS 2019-12-11T16:31:29.392 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-11T16:31:29.392 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=1b78fd04fb1980223b57f0a3f644592e4e26b518 2019-12-11T16:31:29.537 INFO:teuthology.orchestra.run.smithi199:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-12-11T16:31:29.681 INFO:teuthology.orchestra.run.smithi199.stdout:15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:29.681 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-7938.g1b78fd0.el7 2019-12-11T16:31:29.681 INFO:teuthology.task.install:The correct ceph version 15.0.0-7938.g1b78fd0 is installed. 2019-12-11T16:31:29.681 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-12-11T16:31:29.683 INFO:teuthology.orchestra.run.smithi079:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-11T16:31:29.800 INFO:teuthology.orchestra.run.smithi199:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-11T16:31:29.921 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-12-11T16:31:29.921 INFO:teuthology.orchestra.run.smithi079:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-11T16:31:30.045 INFO:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-11T16:31:30.132 INFO:teuthology.orchestra.run.smithi199:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-11T16:31:30.258 INFO:teuthology.orchestra.run.smithi199:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-11T16:31:30.363 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-12-11T16:31:30.363 INFO:teuthology.orchestra.run.smithi079:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-11T16:31:30.476 INFO:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-11T16:31:30.576 INFO:teuthology.orchestra.run.smithi199:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-11T16:31:30.699 INFO:teuthology.orchestra.run.smithi199:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-11T16:31:30.810 INFO:teuthology.run_tasks:Running task ceph... 2019-12-11T16:31:30.908 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-12-11T16:31:30.908 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2019-12-11T16:31:30.913 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 777 /var/log/ceph 2019-12-11T16:31:31.031 INFO:tasks.ceph:Disabling ceph logrotate... 2019-12-11T16:31:31.031 INFO:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-11T16:31:31.035 INFO:teuthology.orchestra.run.smithi199:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-11T16:31:31.141 INFO:tasks.ceph:Creating extra log directories... 2019-12-11T16:31:31.142 INFO:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-11T16:31:31.144 INFO:teuthology.orchestra.run.smithi199:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-11T16:31:31.255 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-12-11T16:31:31.256 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'global': {'osd heartbeat grace': 40, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'mon client directed command retry': 5, 'osd_min_pg_log_entries': 10}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd fast shutdown': False, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'client': {'debug rgw': 20, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt require ssl': False, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10}}} 2019-12-11T16:31:31.256 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'flavor': 'notcmalloc', 'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518'}}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'rgw/test_rgw_gc_log.sh']}}}, {'ragweed': {'client.0': {'rgw_server': 'client.0', 'stages': 'prepare,check', 'default-branch': 'ceph-master'}}}, {'s3tests-java': {'client.0': {'force-repo': 'https://github.com/ceph/java_s3tests.git', 'force-branch': 'master'}}}, {'s3tests': {'client.0': {'rgw_server': 'client.0', 'force-branch': 'public-access-qa', 'git_remote': 'git://github.com/theanalyst/'}}}, {'swift': {'client.0': {'rgw_server': 'client.0'}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-10-1759/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.13073', 'owner': 'scheduled_abhi@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'targets': {'smithi079.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZGRxxko5Y8zu5VAwr2MZuVC5fMBMybUCjvady+TahEihj8hVnQvx5rPKmqqklR6f2c7CYzISj2jSHle4wqQVgT4mkjLGEPZq5EXE4cHJSkjk2ngXOD5mD0ehhF8XQq+3zhhLY2P/JSw+aEBejmT4gxcrv9o98nwjMWEuT5Ks7o2q0Oi9Xaiy4GNtV0ST4BvUOx5qMQi0c6Uxz9k2y+rEk+9OEPnbbgyQlzYPsD0pHfBulfcBVkx2GLjgSrqDV1k7p6dKwj2vGSCNWDi+ve7rSfLo4bQvUekuKA+F2V987PwIlJhn/w8E5qrFm9ocq1R/u/+Y8O1kjcFBVZnM00E6r', 'smithi199.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRzwD4606ZRlVlNgyRMRmn6nTnSbBQDP9absvlSeLbl1jXur1CG+cMVJQ4AgfWt7lz63z1hFkAFkbETWRJJNNI3X8ML8g/wbR1Myxw+a164PNCqoR+jswwdyh6fGID3R1In9ckQgRIhzQMANFbRKy2fkqB6yqhp2F6tI75hCwBEKF5OKWb7XTxmEe5qdyRMgJmCrObCPuyOQTiJWbypBGKOGMfeTfdBbYaO/Xo6fJESoIVnkiUwBaTZvrqtDTo720WJJjYh0JeMhnFYnB5p4OcLCZaz3wUgwhl2lz8c6yhIdacEV6cb0apwCvBfmGXWUJGMYpjYqWW0mXR7RlCNNhf'}, 'job_id': '4593208', 'tube': 'smithi', 'suite_branch': 'wip-abhi-testing-2019-12-10-1759', 'priority': 99, 'first_in_suite': False, 'branch': 'wip-abhi-testing-2019-12-10-1759', 'suite': 'rgw', 'email': None, 'archive_path': '/home/teuthworker/archive/abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi/4593208', 'description': 'rgw/verify/{clusters/fixed-2.yaml frontend/civetweb.yaml msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml overrides.yaml proto/http.yaml rgw_pool_type/ec-profile.yaml tasks/{0-install.yaml cls.yaml ragweed.yaml s3tests-java.yaml s3tests.yaml swift.yaml} validater/valgrind.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518', 'name': 'abhi-2019-12-11_15:51:19-rgw-wip-abhi-testing-2019-12-10-1759-distro-basic-smithi', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518', 'os_type': 'centos', 'overrides': {'s3tests': {'slow_backend': True}, 'ceph-deploy': {'fs': 'xfs', 'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'mon osd full ratio': 0.9, 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'mon osd nearfull ratio': 0.8, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'bluestore block size': 96636764160, 'debug rocksdb': 10, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95, 'bdev async discard': True}}}, 'rgw': {'frontend': 'civetweb', 'compression type': 'random', 'storage classes': 'LUKEWARM, FROZEN', 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 3, 'm': 1, 'name': 'testprofile'}, 'ec-data-pool': True}, 'workunit': {'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518', 'branch': 'wip-abhi-testing-2019-12-10-1759'}, 'ceph': {'conf': {'global': {'osd heartbeat grace': 40, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'mon client directed command retry': 5, 'osd_min_pg_log_entries': 10}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd fast shutdown': False, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'client': {'debug rgw': 20, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt require ssl': False, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}, 'fs': 'xfs', 'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518'}, 'install': {'ceph': {'flavor': 'notcmalloc', 'sha1': '1b78fd04fb1980223b57f0a3f644592e4e26b518'}}, 'admin_socket': {'branch': 'wip-abhi-testing-2019-12-10-1759'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}}} 2019-12-11T16:31:31.256 INFO:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-11T16:31:31.263 INFO:teuthology.orchestra.run.smithi199:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-11T16:31:31.341 INFO:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2019-12-11T16:31:31.345 INFO:teuthology.orchestra.run.smithi199:> sudo install -d -m0777 -- /var/run/ceph 2019-12-11T16:31:31.527 DEBUG:teuthology.orchestra.remote:smithi079:/scratch_devs is 72B 2019-12-11T16:31:31.680 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-11T16:31:31.680 INFO:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-11T16:31:31.784 INFO:teuthology.orchestra.run.smithi079.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-12-11T16:31:31.784 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:31.784 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 98011 Links: 1 2019-12-11T16:31:31.785 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:31.785 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:31.785 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2019-12-11 16:30:18.621919466 +0000 2019-12-11T16:31:31.785 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2019-12-11 16:28:27.281750588 +0000 2019-12-11T16:31:31.785 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2019-12-11 16:28:27.281750588 +0000 2019-12-11T16:31:31.785 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2019-12-11T16:31:31.810 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2019-12-11T16:31:31.810 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2019-12-11T16:31:31.810 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes (512 B) copied, 0.000390676 s, 1.3 MB/s 2019-12-11T16:31:31.814 INFO:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-11T16:31:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-12-11T16:31:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:31.889 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 101070 Links: 1 2019-12-11T16:31:31.889 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:31.889 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:31.889 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2019-12-11 16:30:18.622919466 +0000 2019-12-11T16:31:31.890 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2019-12-11 16:28:27.729740428 +0000 2019-12-11T16:31:31.890 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2019-12-11 16:28:27.729740428 +0000 2019-12-11T16:31:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2019-12-11T16:31:31.913 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2019-12-11T16:31:31.913 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2019-12-11T16:31:31.913 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes (512 B) copied, 0.000319801 s, 1.6 MB/s 2019-12-11T16:31:31.917 INFO:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-11T16:31:31.990 INFO:teuthology.orchestra.run.smithi079.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-12-11T16:31:31.991 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:31.991 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 110640 Links: 1 2019-12-11T16:31:31.991 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:31.991 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:31.992 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2019-12-11 16:30:18.622919466 +0000 2019-12-11T16:31:31.992 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2019-12-11 16:28:28.170730426 +0000 2019-12-11T16:31:31.992 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2019-12-11 16:28:28.170730426 +0000 2019-12-11T16:31:31.992 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2019-12-11T16:31:32.001 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2019-12-11T16:31:32.001 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2019-12-11T16:31:32.001 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes (512 B) copied, 0.000297906 s, 1.7 MB/s 2019-12-11T16:31:32.004 INFO:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-11T16:31:32.057 INFO:teuthology.orchestra.run.smithi079.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-12-11T16:31:32.057 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:32.057 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 98092 Links: 1 2019-12-11T16:31:32.057 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:32.057 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:32.057 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2019-12-11 16:30:18.622919466 +0000 2019-12-11T16:31:32.058 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2019-12-11 16:28:28.642719722 +0000 2019-12-11T16:31:32.058 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2019-12-11 16:28:28.642719722 +0000 2019-12-11T16:31:32.058 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2019-12-11T16:31:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2019-12-11T16:31:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2019-12-11T16:31:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes (512 B) copied, 0.000293699 s, 1.7 MB/s 2019-12-11T16:31:32.075 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-11T16:31:32.075 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-11T16:31:32.076 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-11T16:31:32.076 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:32.169 DEBUG:teuthology.orchestra.remote:smithi199:/scratch_devs is 72B 2019-12-11T16:31:32.276 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-11T16:31:32.277 INFO:teuthology.orchestra.run.smithi199:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-11T16:31:32.361 INFO:teuthology.orchestra.run.smithi199.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-12-11T16:31:32.361 INFO:teuthology.orchestra.run.smithi199.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:32.361 INFO:teuthology.orchestra.run.smithi199.stdout:Device: 5h/5d Inode: 107767 Links: 1 2019-12-11T16:31:32.362 INFO:teuthology.orchestra.run.smithi199.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:32.362 INFO:teuthology.orchestra.run.smithi199.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:32.362 INFO:teuthology.orchestra.run.smithi199.stdout:Access: 2019-12-11 16:30:18.505200349 +0000 2019-12-11T16:31:32.362 INFO:teuthology.orchestra.run.smithi199.stdout:Modify: 2019-12-11 16:28:27.232519839 +0000 2019-12-11T16:31:32.362 INFO:teuthology.orchestra.run.smithi199.stdout:Change: 2019-12-11 16:28:27.232519839 +0000 2019-12-11T16:31:32.362 INFO:teuthology.orchestra.run.smithi199.stdout: Birth: - 2019-12-11T16:31:32.388 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records in 2019-12-11T16:31:32.388 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records out 2019-12-11T16:31:32.388 INFO:teuthology.orchestra.run.smithi199.stderr:512 bytes (512 B) copied, 0.000479534 s, 1.1 MB/s 2019-12-11T16:31:32.394 INFO:teuthology.orchestra.run.smithi199:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-11T16:31:32.463 INFO:teuthology.orchestra.run.smithi199.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-12-11T16:31:32.464 INFO:teuthology.orchestra.run.smithi199.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:32.464 INFO:teuthology.orchestra.run.smithi199.stdout:Device: 5h/5d Inode: 110717 Links: 1 2019-12-11T16:31:32.464 INFO:teuthology.orchestra.run.smithi199.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:32.464 INFO:teuthology.orchestra.run.smithi199.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:32.464 INFO:teuthology.orchestra.run.smithi199.stdout:Access: 2019-12-11 16:30:18.505200349 +0000 2019-12-11T16:31:32.464 INFO:teuthology.orchestra.run.smithi199.stdout:Modify: 2019-12-11 16:28:27.693512456 +0000 2019-12-11T16:31:32.465 INFO:teuthology.orchestra.run.smithi199.stdout:Change: 2019-12-11 16:28:27.693512456 +0000 2019-12-11T16:31:32.465 INFO:teuthology.orchestra.run.smithi199.stdout: Birth: - 2019-12-11T16:31:32.490 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records in 2019-12-11T16:31:32.491 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records out 2019-12-11T16:31:32.491 INFO:teuthology.orchestra.run.smithi199.stderr:512 bytes (512 B) copied, 0.000408107 s, 1.3 MB/s 2019-12-11T16:31:32.495 INFO:teuthology.orchestra.run.smithi199:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-11T16:31:32.567 INFO:teuthology.orchestra.run.smithi199.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-12-11T16:31:32.567 INFO:teuthology.orchestra.run.smithi199.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:32.567 INFO:teuthology.orchestra.run.smithi199.stdout:Device: 5h/5d Inode: 102977 Links: 1 2019-12-11T16:31:32.567 INFO:teuthology.orchestra.run.smithi199.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:32.568 INFO:teuthology.orchestra.run.smithi199.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:32.568 INFO:teuthology.orchestra.run.smithi199.stdout:Access: 2019-12-11 16:30:18.505200349 +0000 2019-12-11T16:31:32.568 INFO:teuthology.orchestra.run.smithi199.stdout:Modify: 2019-12-11 16:28:28.145505218 +0000 2019-12-11T16:31:32.568 INFO:teuthology.orchestra.run.smithi199.stdout:Change: 2019-12-11 16:28:28.145505218 +0000 2019-12-11T16:31:32.568 INFO:teuthology.orchestra.run.smithi199.stdout: Birth: - 2019-12-11T16:31:32.593 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records in 2019-12-11T16:31:32.593 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records out 2019-12-11T16:31:32.594 INFO:teuthology.orchestra.run.smithi199.stderr:512 bytes (512 B) copied, 0.000407195 s, 1.3 MB/s 2019-12-11T16:31:32.598 INFO:teuthology.orchestra.run.smithi199:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-11T16:31:32.659 INFO:teuthology.orchestra.run.smithi199.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-12-11T16:31:32.660 INFO:teuthology.orchestra.run.smithi199.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-11T16:31:32.660 INFO:teuthology.orchestra.run.smithi199.stdout:Device: 5h/5d Inode: 108620 Links: 1 2019-12-11T16:31:32.660 INFO:teuthology.orchestra.run.smithi199.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-11T16:31:32.660 INFO:teuthology.orchestra.run.smithi199.stdout:Context: system_u:object_r:device_t:s0 2019-12-11T16:31:32.660 INFO:teuthology.orchestra.run.smithi199.stdout:Access: 2019-12-11 16:30:18.505200349 +0000 2019-12-11T16:31:32.660 INFO:teuthology.orchestra.run.smithi199.stdout:Modify: 2019-12-11 16:28:28.630497451 +0000 2019-12-11T16:31:32.660 INFO:teuthology.orchestra.run.smithi199.stdout:Change: 2019-12-11 16:28:28.630497451 +0000 2019-12-11T16:31:32.672 INFO:teuthology.orchestra.run.smithi199.stdout: Birth: - 2019-12-11T16:31:32.680 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records in 2019-12-11T16:31:32.680 INFO:teuthology.orchestra.run.smithi199.stderr:1+0 records out 2019-12-11T16:31:32.681 INFO:teuthology.orchestra.run.smithi199.stderr:512 bytes (512 B) copied, 0.000325578 s, 1.6 MB/s 2019-12-11T16:31:32.683 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-11T16:31:32.683 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-11T16:31:32.683 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-11T16:31:32.683 INFO:tasks.ceph:dev map: {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:32.683 INFO:tasks.ceph:Generating config... 2019-12-11T16:31:32.685 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2019-12-11T16:31:32.685 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2019-12-11T16:31:32.685 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2019-12-11T16:31:32.685 INFO:tasks.ceph:[global] mon client directed command retry = 5 2019-12-11T16:31:32.685 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2019-12-11T16:31:32.685 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-12-11T16:31:32.686 INFO:tasks.ceph:[osd] debug ms = 1 2019-12-11T16:31:32.686 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-12-11T16:31:32.686 INFO:tasks.ceph:[osd] debug osd = 25 2019-12-11T16:31:32.686 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-12-11T16:31:32.686 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-12-11T16:31:32.686 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-12-11T16:31:32.686 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] debug journal = 20 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] bdev async discard = True 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] osd fast shutdown = False 2019-12-11T16:31:32.687 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-12-11T16:31:32.688 INFO:tasks.ceph:[osd] debug filestore = 20 2019-12-11T16:31:32.688 INFO:tasks.ceph:[osd] bdev enable discard = True 2019-12-11T16:31:32.688 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-12-11T16:31:32.690 INFO:tasks.ceph:[mon] debug mon = 20 2019-12-11T16:31:32.690 INFO:tasks.ceph:[mon] debug paxos = 20 2019-12-11T16:31:32.691 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2019-12-11T16:31:32.691 INFO:tasks.ceph:[mon] debug ms = 1 2019-12-11T16:31:32.691 INFO:tasks.ceph:[client] debug rgw = 20 2019-12-11T16:31:32.691 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2019-12-11T16:31:32.691 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2019-12-11T16:31:32.691 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2019-12-11T16:31:32.691 INFO:tasks.ceph:[client] rgw lc debug interval = 10 2019-12-11T16:31:32.692 INFO:tasks.ceph:Setting up mon.a... 2019-12-11T16:31:32.692 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-12-11T16:31:32.858 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.keyring 2019-12-11T16:31:32.862 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-12-11T16:31:33.004 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-11T16:31:33.118 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.79'), ('mon.c', '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]'), ('mon.b', '172.21.15.199')] 2019-12-11T16:31:33.118 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.79 --addv c '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]' --add b 172.21.15.199 --print /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:33.258 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:33.258 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: generated fsid b3142dac-fc19-47e9-ad13-f81d4b6261ad 2019-12-11T16:31:33.258 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 0 2019-12-11T16:31:33.258 INFO:teuthology.orchestra.run.smithi079.stdout:fsid b3142dac-fc19-47e9-ad13-f81d4b6261ad 2019-12-11T16:31:33.258 INFO:teuthology.orchestra.run.smithi079.stdout:last_changed 2019-12-11T16:31:33.260677+0000 2019-12-11T16:31:33.259 INFO:teuthology.orchestra.run.smithi079.stdout:created 2019-12-11T16:31:33.260677+0000 2019-12-11T16:31:33.259 INFO:teuthology.orchestra.run.smithi079.stdout:min_mon_release 0 (unknown) 2019-12-11T16:31:33.259 INFO:teuthology.orchestra.run.smithi079.stdout:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2019-12-11T16:31:33.259 INFO:teuthology.orchestra.run.smithi079.stdout:1: [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] mon.b 2019-12-11T16:31:33.259 INFO:teuthology.orchestra.run.smithi079.stdout:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2019-12-11T16:31:33.259 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-12-11T16:31:33.260 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b3142dac-fc19-47e9-ad13-f81d4b6261ad... 2019-12-11T16:31:33.261 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-11T16:31:33.265 INFO:teuthology.orchestra.run.smithi199:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-11T16:31:33.488 INFO:tasks.ceph:Creating admin key on mon.a... 2019-12-11T16:31:33.488 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-12-11T16:31:33.663 INFO:tasks.ceph:Copying monmap to all nodes... 2019-12-11T16:31:33.752 DEBUG:teuthology.orchestra.remote:smithi079:/etc/ceph/ceph.keyring is 206B 2019-12-11T16:31:33.897 DEBUG:teuthology.orchestra.remote:smithi079:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-12-11T16:31:33.964 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:31:33.964 INFO:teuthology.orchestra.run.smithi079:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-11T16:31:34.097 INFO:teuthology.orchestra.run.smithi079:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:34.168 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:31:34.169 INFO:teuthology.orchestra.run.smithi199:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-11T16:31:34.341 INFO:teuthology.orchestra.run.smithi199:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:34.429 INFO:tasks.ceph:Setting up mon nodes... 2019-12-11T16:31:34.429 INFO:tasks.ceph:Setting up mgr nodes... 2019-12-11T16:31:34.429 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-12-11T16:31:34.604 INFO:teuthology.orchestra.run.smithi079.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-12-11T16:31:34.606 INFO:teuthology.orchestra.run.smithi199:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-12-11T16:31:34.825 INFO:teuthology.orchestra.run.smithi199.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-12-11T16:31:34.829 INFO:tasks.ceph:Setting up mds nodes... 2019-12-11T16:31:34.830 INFO:tasks.ceph_client:Setting up client nodes... 2019-12-11T16:31:34.830 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-12-11T16:31:34.975 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-12-11T16:31:35.006 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2019-12-11T16:31:35.156 INFO:teuthology.orchestra.run.smithi199.stdout:creating /etc/ceph/ceph.client.1.keyring 2019-12-11T16:31:35.187 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-12-11T16:31:35.187 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi199.front.sepia.ceph.com'): {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'}} 2019-12-11T16:31:35.187 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-12-11T16:31:35.304 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:35.304 INFO:tasks.ceph:{} 2019-12-11T16:31:35.305 INFO:tasks.ceph:osd.0 2019-12-11T16:31:35.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:31:35.305 INFO:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-11T16:31:35.862 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:35.863 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:35.863 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:35.863 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:35.863 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:35.863 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:35.863 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:35.864 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:35.864 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:35.865 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2019-12-11T16:31:35.865 INFO:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2019-12-11T16:31:36.037 INFO:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2019-12-11T16:31:36.158 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-12-11T16:31:36.240 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:36.240 INFO:tasks.ceph:{} 2019-12-11T16:31:36.240 INFO:tasks.ceph:osd.1 2019-12-11T16:31:36.240 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:31:36.240 INFO:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-11T16:31:36.763 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:36.763 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:36.763 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:36.763 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:36.764 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:36.764 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:36.764 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:36.764 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:36.764 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:36.766 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2019-12-11T16:31:36.767 INFO:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2019-12-11T16:31:36.926 INFO:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2019-12-11T16:31:37.040 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-12-11T16:31:37.120 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:37.120 INFO:tasks.ceph:{} 2019-12-11T16:31:37.121 INFO:tasks.ceph:osd.2 2019-12-11T16:31:37.121 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:31:37.121 INFO:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-11T16:31:37.733 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:37.733 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:37.733 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:37.733 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:37.734 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:37.734 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:37.734 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:37.734 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:37.734 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:37.736 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2019-12-11T16:31:37.736 INFO:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2019-12-11T16:31:37.903 INFO:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2019-12-11T16:31:38.025 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-12-11T16:31:38.122 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:38.123 INFO:tasks.ceph:{} 2019-12-11T16:31:38.123 INFO:tasks.ceph:osd.3 2019-12-11T16:31:38.123 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:31:38.124 INFO:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-11T16:31:38.696 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:38.696 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:38.696 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:38.696 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:38.696 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:38.696 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:38.697 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:38.697 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:38.697 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:38.698 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2019-12-11T16:31:38.699 INFO:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2019-12-11T16:31:38.847 INFO:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2019-12-11T16:31:38.964 INFO:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:39.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:39.108+0000 7fae1b967b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-12-11T16:31:39.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:39.109+0000 7fae1b967b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-12-11T16:31:39.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:39.109+0000 7fae1b967b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-11T16:31:39.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:39.109+0000 7fae1b967b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-11T16:31:39.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:39.109+0000 7fae1b967b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-11T16:31:39.114 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:39.115+0000 7fae1b967b40 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-12-11T16:31:41.425 INFO:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-12-11T16:31:41.543 INFO:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:41.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:41.688+0000 7fdb872afb40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-12-11T16:31:41.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:41.689+0000 7fdb872afb40 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-12-11T16:31:41.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:41.689+0000 7fdb872afb40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-11T16:31:41.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:41.689+0000 7fdb872afb40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-11T16:31:41.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:41.689+0000 7fdb872afb40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-11T16:31:41.691 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:41.690+0000 7fdb872afb40 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-12-11T16:31:43.997 INFO:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-12-11T16:31:44.121 INFO:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:44.238+0000 7fa8a5467b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-12-11T16:31:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:44.238+0000 7fa8a5467b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-12-11T16:31:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:44.238+0000 7fa8a5467b40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-11T16:31:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:44.239+0000 7fa8a5467b40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-11T16:31:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:44.239+0000 7fa8a5467b40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-11T16:31:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:44.239+0000 7fa8a5467b40 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-12-11T16:31:46.545 INFO:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-12-11T16:31:46.662 INFO:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:46.808 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:46.809+0000 7f53d4d0ab40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-11T16:31:46.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:46.810+0000 7f53d4d0ab40 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-12-11T16:31:46.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:46.810+0000 7f53d4d0ab40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-11T16:31:46.810 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:46.810+0000 7f53d4d0ab40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-11T16:31:46.810 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:46.810+0000 7f53d4d0ab40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-11T16:31:46.810 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:31:46.811+0000 7f53d4d0ab40 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-12-11T16:31:49.122 INFO:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-12-11T16:31:49.240 INFO:teuthology.orchestra.run.smithi199:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-12-11T16:31:49.361 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:49.361 INFO:tasks.ceph:{} 2019-12-11T16:31:49.361 INFO:tasks.ceph:osd.4 2019-12-11T16:31:49.362 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:31:49.362 INFO:teuthology.orchestra.run.smithi199:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-11T16:31:49.946 INFO:teuthology.orchestra.run.smithi199.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:49.946 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:49.946 INFO:teuthology.orchestra.run.smithi199.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:49.946 INFO:teuthology.orchestra.run.smithi199.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:49.946 INFO:teuthology.orchestra.run.smithi199.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:49.947 INFO:teuthology.orchestra.run.smithi199.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:49.947 INFO:teuthology.orchestra.run.smithi199.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:49.947 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:49.947 INFO:teuthology.orchestra.run.smithi199.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:49.949 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi199.front.sepia.ceph.com -o noatime 2019-12-11T16:31:49.949 INFO:teuthology.orchestra.run.smithi199:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4 2019-12-11T16:31:50.113 INFO:teuthology.orchestra.run.smithi199:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2019-12-11T16:31:50.236 INFO:teuthology.orchestra.run.smithi199:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-12-11T16:31:50.348 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:50.348 INFO:tasks.ceph:{} 2019-12-11T16:31:50.348 INFO:tasks.ceph:osd.5 2019-12-11T16:31:50.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:31:50.349 INFO:teuthology.orchestra.run.smithi199:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-11T16:31:50.907 INFO:teuthology.orchestra.run.smithi199.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:50.907 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:50.907 INFO:teuthology.orchestra.run.smithi199.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:50.907 INFO:teuthology.orchestra.run.smithi199.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:50.908 INFO:teuthology.orchestra.run.smithi199.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:50.908 INFO:teuthology.orchestra.run.smithi199.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:50.908 INFO:teuthology.orchestra.run.smithi199.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:50.908 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:50.908 INFO:teuthology.orchestra.run.smithi199.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:50.910 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi199.front.sepia.ceph.com -o noatime 2019-12-11T16:31:50.910 INFO:teuthology.orchestra.run.smithi199:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2019-12-11T16:31:51.068 INFO:teuthology.orchestra.run.smithi199:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2019-12-11T16:31:51.185 INFO:teuthology.orchestra.run.smithi199:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-12-11T16:31:51.280 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:51.280 INFO:tasks.ceph:{} 2019-12-11T16:31:51.281 INFO:tasks.ceph:osd.6 2019-12-11T16:31:51.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:31:51.281 INFO:teuthology.orchestra.run.smithi199:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-11T16:31:51.843 INFO:teuthology.orchestra.run.smithi199.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:51.843 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:51.843 INFO:teuthology.orchestra.run.smithi199.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:51.843 INFO:teuthology.orchestra.run.smithi199.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:51.843 INFO:teuthology.orchestra.run.smithi199.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:51.844 INFO:teuthology.orchestra.run.smithi199.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:51.844 INFO:teuthology.orchestra.run.smithi199.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:51.844 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:51.844 INFO:teuthology.orchestra.run.smithi199.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:51.845 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi199.front.sepia.ceph.com -o noatime 2019-12-11T16:31:51.846 INFO:teuthology.orchestra.run.smithi199:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2019-12-11T16:31:52.029 INFO:teuthology.orchestra.run.smithi199:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2019-12-11T16:31:52.166 INFO:teuthology.orchestra.run.smithi199:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-12-11T16:31:52.253 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-11T16:31:52.253 INFO:tasks.ceph:{} 2019-12-11T16:31:52.253 INFO:tasks.ceph:osd.7 2019-12-11T16:31:52.254 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:31:52.254 INFO:teuthology.orchestra.run.smithi199:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-11T16:31:52.768 INFO:teuthology.orchestra.run.smithi199.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-11T16:31:52.769 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-11T16:31:52.769 INFO:teuthology.orchestra.run.smithi199.stdout: = crc=1 finobt=0, sparse=0 2019-12-11T16:31:52.770 INFO:teuthology.orchestra.run.smithi199.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-11T16:31:52.770 INFO:teuthology.orchestra.run.smithi199.stdout: = sunit=0 swidth=0 blks 2019-12-11T16:31:52.770 INFO:teuthology.orchestra.run.smithi199.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-11T16:31:52.770 INFO:teuthology.orchestra.run.smithi199.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-11T16:31:52.770 INFO:teuthology.orchestra.run.smithi199.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-11T16:31:52.770 INFO:teuthology.orchestra.run.smithi199.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-11T16:31:52.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi199.front.sepia.ceph.com -o noatime 2019-12-11T16:31:52.771 INFO:teuthology.orchestra.run.smithi199:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-7 2019-12-11T16:31:52.901 INFO:teuthology.orchestra.run.smithi199:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2019-12-11T16:31:53.001 INFO:teuthology.orchestra.run.smithi199:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:53.171 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:53.164+0000 7fd3ce534b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-12-11T16:31:53.173 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:53.166+0000 7fd3ce534b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-12-11T16:31:53.173 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:53.166+0000 7fd3ce534b40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-11T16:31:53.173 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:53.166+0000 7fd3ce534b40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-11T16:31:53.173 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:53.166+0000 7fd3ce534b40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-11T16:31:53.177 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:53.171+0000 7fd3ce534b40 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-12-11T16:31:55.487 INFO:teuthology.orchestra.run.smithi199:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-12-11T16:31:55.600 INFO:teuthology.orchestra.run.smithi199:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:55.746 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:55.746+0000 7f11b2519b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-12-11T16:31:55.746 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:55.747+0000 7f11b2519b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-12-11T16:31:55.747 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:55.747+0000 7f11b2519b40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-11T16:31:55.747 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:55.747+0000 7f11b2519b40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-11T16:31:55.747 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:55.747+0000 7f11b2519b40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-11T16:31:55.747 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:55.748+0000 7f11b2519b40 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-12-11T16:31:58.059 INFO:teuthology.orchestra.run.smithi199:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-12-11T16:31:58.179 INFO:teuthology.orchestra.run.smithi199:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:31:58.322 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:58.322+0000 7f36a5ef1b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-12-11T16:31:58.322 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:58.323+0000 7f36a5ef1b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-12-11T16:31:58.322 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:58.323+0000 7f36a5ef1b40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-11T16:31:58.323 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:58.323+0000 7f36a5ef1b40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-11T16:31:58.323 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:58.323+0000 7f36a5ef1b40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-11T16:31:58.323 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:31:58.324+0000 7f36a5ef1b40 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2019-12-11T16:32:00.635 INFO:teuthology.orchestra.run.smithi199:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-12-11T16:32:00.751 INFO:teuthology.orchestra.run.smithi199:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:32:00.891 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:32:00.892+0000 7f921ed5bb40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-12-11T16:32:00.892 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:32:00.893+0000 7f921ed5bb40 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-12-11T16:32:00.892 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:32:00.893+0000 7f921ed5bb40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-11T16:32:00.892 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:32:00.893+0000 7f921ed5bb40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-11T16:32:00.892 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:32:00.893+0000 7f921ed5bb40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-11T16:32:00.893 INFO:teuthology.orchestra.run.smithi199.stderr:2019-12-11T16:32:00.894+0000 7f921ed5bb40 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2019-12-11T16:32:03.204 INFO:teuthology.orchestra.run.smithi199:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-12-11T16:32:03.323 INFO:tasks.ceph:Reading keys from all nodes... 2019-12-11T16:32:03.324 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:03.415 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.29WR5Wo3HO 2019-12-11T16:32:03.416 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.29WR5Wo3HO 2019-12-11T16:32:03.537 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.29WR5Wo3HO 2019-12-11T16:32:03.792 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.29WR5Wo3HO is 56B 2019-12-11T16:32:03.892 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.29WR5Wo3HO 2019-12-11T16:32:03.986 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:04.060 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.0qbuMPqRje 2019-12-11T16:32:04.061 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.0qbuMPqRje 2019-12-11T16:32:04.163 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.0qbuMPqRje 2019-12-11T16:32:04.304 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.0qbuMPqRje is 56B 2019-12-11T16:32:04.363 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.0qbuMPqRje 2019-12-11T16:32:04.419 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:04.474 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.9jl7capf5P 2019-12-11T16:32:04.475 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.9jl7capf5P 2019-12-11T16:32:04.544 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.9jl7capf5P 2019-12-11T16:32:04.686 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.9jl7capf5P is 56B 2019-12-11T16:32:04.748 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.9jl7capf5P 2019-12-11T16:32:04.798 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:04.915 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.TC8e3k1Vtx 2019-12-11T16:32:04.916 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.TC8e3k1Vtx 2019-12-11T16:32:05.015 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.TC8e3k1Vtx 2019-12-11T16:32:05.165 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.TC8e3k1Vtx is 56B 2019-12-11T16:32:05.231 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.TC8e3k1Vtx 2019-12-11T16:32:05.287 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:05.414 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.IU2q6NqeYD 2019-12-11T16:32:05.414 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.IU2q6NqeYD 2019-12-11T16:32:05.530 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.IU2q6NqeYD 2019-12-11T16:32:05.729 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.IU2q6NqeYD is 56B 2019-12-11T16:32:05.818 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.IU2q6NqeYD 2019-12-11T16:32:05.901 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:05.988 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.rDGqag7Vsa 2019-12-11T16:32:05.989 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.rDGqag7Vsa 2019-12-11T16:32:06.107 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.rDGqag7Vsa 2019-12-11T16:32:06.323 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.rDGqag7Vsa is 56B 2019-12-11T16:32:06.446 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.rDGqag7Vsa 2019-12-11T16:32:06.543 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:06.622 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.FPaGzDGMxI 2019-12-11T16:32:06.622 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.FPaGzDGMxI 2019-12-11T16:32:06.737 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.FPaGzDGMxI 2019-12-11T16:32:06.959 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.FPaGzDGMxI is 56B 2019-12-11T16:32:07.059 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.FPaGzDGMxI 2019-12-11T16:32:07.139 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:07.225 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.Ty8K0k18OG 2019-12-11T16:32:07.225 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.Ty8K0k18OG 2019-12-11T16:32:07.312 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.Ty8K0k18OG 2019-12-11T16:32:07.450 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.Ty8K0k18OG is 56B 2019-12-11T16:32:07.514 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.Ty8K0k18OG 2019-12-11T16:32:07.565 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:07.694 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.uvAsYUyVsm 2019-12-11T16:32:07.695 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.uvAsYUyVsm 2019-12-11T16:32:07.793 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.uvAsYUyVsm 2019-12-11T16:32:07.993 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.uvAsYUyVsm is 56B 2019-12-11T16:32:08.072 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.uvAsYUyVsm 2019-12-11T16:32:08.146 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:08.273 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.XBwCacWgYw 2019-12-11T16:32:08.274 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.XBwCacWgYw 2019-12-11T16:32:08.382 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.XBwCacWgYw 2019-12-11T16:32:08.542 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.XBwCacWgYw is 56B 2019-12-11T16:32:08.619 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.XBwCacWgYw 2019-12-11T16:32:08.792 DEBUG:teuthology.orchestra.remote:smithi079:/etc/ceph/ceph.client.0.keyring is 59B 2019-12-11T16:32:08.979 DEBUG:teuthology.orchestra.remote:smithi199:/etc/ceph/ceph.client.1.keyring is 59B 2019-12-11T16:32:09.073 INFO:tasks.ceph:Adding keys to all mons... 2019-12-11T16:32:09.073 INFO:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-11T16:32:09.088 INFO:teuthology.orchestra.run.smithi199:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-11T16:32:09.189 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.y] 2019-12-11T16:32:09.189 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDmGfFdihUDJBAAIRIaPlORDAGNpGXSZZr7Jw== 2019-12-11T16:32:09.189 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2019-12-11T16:32:09.190 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDrGfFd33aSBhAAQCllW6HQ8LA11YFi0oywmw== 2019-12-11T16:32:09.190 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2019-12-11T16:32:09.190 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDtGfFd4jQlKRAAJ1Eq9d7HJe8Zll5eov4Yuw== 2019-12-11T16:32:09.190 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2019-12-11T16:32:09.190 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDwGfFdIKhRDhAAujODyWVSH5t/1BMv8RtDRg== 2019-12-11T16:32:09.190 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2019-12-11T16:32:09.190 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDyGfFdVZRkMBAAxRuLi7vIbIo6BfXZ/UMkXg== 2019-12-11T16:32:09.191 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2019-12-11T16:32:09.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDmGfFd/mtIMRAAf4jj/9yHqokpA4rYdvR+Ew== 2019-12-11T16:32:09.191 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.4] 2019-12-11T16:32:09.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQD5GfFdaVvcCRAAgQ30HMQoYnoRc53BxacVMg== 2019-12-11T16:32:09.191 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.5] 2019-12-11T16:32:09.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQD7GfFdGFCXLBAAHtOAYtCQkYskI5n4hTe/gA== 2019-12-11T16:32:09.192 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.6] 2019-12-11T16:32:09.192 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQD+GfFdCzhRExAAKLH0FnjLPNCakpVPJxT08Q== 2019-12-11T16:32:09.192 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.7] 2019-12-11T16:32:09.192 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAAGvFdIChENRAA6KumUxAcgBQiXEF0Priyfw== 2019-12-11T16:32:09.192 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2019-12-11T16:32:09.192 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDmGfFdRR5LOhAAzYYQqP+imI63sGhS+cOajQ== 2019-12-11T16:32:09.192 INFO:teuthology.orchestra.run.smithi079.stdout:[client.1] 2019-12-11T16:32:09.193 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDnGfFdSglcCRAA0DMYAxhY+40S6BR/l1uLqw== 2019-12-11T16:32:09.200 INFO:teuthology.orchestra.run.smithi199.stdout:[mgr.y] 2019-12-11T16:32:09.201 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDmGfFdihUDJBAAIRIaPlORDAGNpGXSZZr7Jw== 2019-12-11T16:32:09.201 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.0] 2019-12-11T16:32:09.201 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDrGfFd33aSBhAAQCllW6HQ8LA11YFi0oywmw== 2019-12-11T16:32:09.201 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.1] 2019-12-11T16:32:09.201 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDtGfFd4jQlKRAAJ1Eq9d7HJe8Zll5eov4Yuw== 2019-12-11T16:32:09.201 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.2] 2019-12-11T16:32:09.202 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDwGfFdIKhRDhAAujODyWVSH5t/1BMv8RtDRg== 2019-12-11T16:32:09.202 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.3] 2019-12-11T16:32:09.202 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDyGfFdVZRkMBAAxRuLi7vIbIo6BfXZ/UMkXg== 2019-12-11T16:32:09.202 INFO:teuthology.orchestra.run.smithi199.stdout:[mgr.x] 2019-12-11T16:32:09.202 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDmGfFd/mtIMRAAf4jj/9yHqokpA4rYdvR+Ew== 2019-12-11T16:32:09.202 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.4] 2019-12-11T16:32:09.202 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQD5GfFdaVvcCRAAgQ30HMQoYnoRc53BxacVMg== 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.5] 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQD7GfFdGFCXLBAAHtOAYtCQkYskI5n4hTe/gA== 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.6] 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQD+GfFdCzhRExAAKLH0FnjLPNCakpVPJxT08Q== 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout:[osd.7] 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQAAGvFdIChENRAA6KumUxAcgBQiXEF0Priyfw== 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout:[client.0] 2019-12-11T16:32:09.203 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDmGfFdRR5LOhAAzYYQqP+imI63sGhS+cOajQ== 2019-12-11T16:32:09.204 INFO:teuthology.orchestra.run.smithi199.stdout:[client.1] 2019-12-11T16:32:09.204 INFO:teuthology.orchestra.run.smithi199.stdout: key = AQDnGfFdSglcCRAA0DMYAxhY+40S6BR/l1uLqw== 2019-12-11T16:32:09.205 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-11T16:32:09.208 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-11T16:32:09.375 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:09.379 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:09.551 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:09.629 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:09.808 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:09.821 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:09.996 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.000 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.170 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-11T16:32:10.173 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-11T16:32:10.338 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.343 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.516 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.521 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.688 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.692 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.863 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:10.867 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-11T16:32:11.034 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-11T16:32:11.038 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-11T16:32:11.208 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-11T16:32:11.212 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-11T16:32:11.380 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-12-11T16:32:11.676 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-12-11T16:32:11.792 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-11T16:32:11.938 INFO:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-12-11T16:32:12.051 INFO:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-12-11T16:32:12.148 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-11T16:32:12.273 INFO:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-12-11T16:32:12.341 INFO:teuthology.orchestra.run.smithi199:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-12-11T16:32:12.465 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-11T16:32:12.640 INFO:teuthology.orchestra.run.smithi199:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-12-11T16:32:12.751 INFO:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:32:12.755 INFO:teuthology.orchestra.run.smithi199:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-11T16:32:12.842 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-12-11T16:32:12.854 DEBUG:teuthology.misc:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2019-12-11T16:32:12.854 INFO:tasks.ceph.mon.a:Restarting daemon 2019-12-11T16:32:12.854 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2019-12-11T16:32:12.868 INFO:tasks.ceph.mon.a:Started 2019-12-11T16:32:12.869 DEBUG:teuthology.misc:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2019-12-11T16:32:12.869 INFO:tasks.ceph.mon.c:Restarting daemon 2019-12-11T16:32:12.869 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2019-12-11T16:32:12.885 INFO:tasks.ceph.mon.c:Started 2019-12-11T16:32:12.885 DEBUG:teuthology.misc:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2019-12-11T16:32:12.885 INFO:tasks.ceph.mon.b:Restarting daemon 2019-12-11T16:32:12.885 INFO:teuthology.orchestra.run.smithi199:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2019-12-11T16:32:12.901 INFO:tasks.ceph.mon.b:Started 2019-12-11T16:32:12.901 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-12-11T16:32:12.901 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-12-11T16:32:12.901 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2019-12-11T16:32:12.909 INFO:tasks.ceph.mgr.y:Started 2019-12-11T16:32:12.910 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-12-11T16:32:12.910 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2019-12-11T16:32:12.919 INFO:tasks.ceph.mgr.x:Started 2019-12-11T16:32:12.920 INFO:tasks.ceph:Setting crush tunables to default 2019-12-11T16:32:12.920 INFO:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd crush tunables default 2019-12-11T16:32:16.747 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:00:03.683 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:32:16.922 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:00:03.829 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:32:16.985 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:00:03.922 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:32:34.032 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2019-12-11T16:32:34.046 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-12-11T16:32:34.046 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:34.091 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.u50lbuNEdG 2019-12-11T16:32:34.092 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.u50lbuNEdG 2019-12-11T16:32:34.193 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.u50lbuNEdG 2019-12-11T16:32:34.292 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.u50lbuNEdG is 37B 2019-12-11T16:32:34.339 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.u50lbuNEdG 2019-12-11T16:32:34.381 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.W8bC6EzKw6 2019-12-11T16:32:34.489 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.W8bC6EzKw6 2019-12-11T16:32:34.545 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.W8bC6EzKw6 2019-12-11T16:32:34.642 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.W8bC6EzKw6 is 37B 2019-12-11T16:32:34.690 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.W8bC6EzKw6 2019-12-11T16:32:34.736 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:34.824 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.KhG9Mb2quA 2019-12-11T16:32:34.824 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.KhG9Mb2quA 2019-12-11T16:32:34.890 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.KhG9Mb2quA 2019-12-11T16:32:35.008 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.KhG9Mb2quA is 37B 2019-12-11T16:32:35.063 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.KhG9Mb2quA 2019-12-11T16:32:35.117 INFO:teuthology.orchestra.run.smithi079:> mktemp 2019-12-11T16:32:35.205 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.cVB7eKawUR 2019-12-11T16:32:35.205 INFO:teuthology.orchestra.run.smithi079:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.cVB7eKawUR 2019-12-11T16:32:35.260 INFO:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.cVB7eKawUR 2019-12-11T16:32:35.418 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.cVB7eKawUR is 37B 2019-12-11T16:32:35.538 INFO:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.cVB7eKawUR 2019-12-11T16:32:35.630 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:35.703 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.2mhVpJwKR7 2019-12-11T16:32:35.703 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.2mhVpJwKR7 2019-12-11T16:32:35.806 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.2mhVpJwKR7 2019-12-11T16:32:35.952 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.2mhVpJwKR7 is 37B 2019-12-11T16:32:36.008 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.2mhVpJwKR7 2019-12-11T16:32:36.062 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:36.153 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.s0U57ccC3d 2019-12-11T16:32:36.153 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.s0U57ccC3d 2019-12-11T16:32:36.220 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.s0U57ccC3d 2019-12-11T16:32:36.329 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.s0U57ccC3d is 37B 2019-12-11T16:32:36.376 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.s0U57ccC3d 2019-12-11T16:32:36.420 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:36.520 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.YpyS4Ro7uz 2019-12-11T16:32:36.520 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.YpyS4Ro7uz 2019-12-11T16:32:36.609 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.YpyS4Ro7uz 2019-12-11T16:32:36.714 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.YpyS4Ro7uz is 37B 2019-12-11T16:32:36.764 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.YpyS4Ro7uz 2019-12-11T16:32:36.809 INFO:teuthology.orchestra.run.smithi199:> mktemp 2019-12-11T16:32:36.921 INFO:teuthology.orchestra.run.smithi199.stdout:/tmp/tmp.WfEhvhgtqr 2019-12-11T16:32:36.921 INFO:teuthology.orchestra.run.smithi199:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.WfEhvhgtqr 2019-12-11T16:32:37.008 INFO:teuthology.orchestra.run.smithi199:> sudo chmod 0666 /tmp/tmp.WfEhvhgtqr 2019-12-11T16:32:37.142 DEBUG:teuthology.orchestra.remote:smithi199:/tmp/tmp.WfEhvhgtqr is 37B 2019-12-11T16:32:37.195 INFO:teuthology.orchestra.run.smithi199:> rm -fr /tmp/tmp.WfEhvhgtqr 2019-12-11T16:32:37.247 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new 1a8edef5-7984-4751-b15c-7c7aac7d8ca3 0 2019-12-11T16:32:38.480 INFO:teuthology.orchestra.run.smithi199.stdout:0 2019-12-11T16:32:38.502 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new a652677c-9e77-4b93-b8d5-16a6b74760ec 1 2019-12-11T16:32:39.741 INFO:teuthology.orchestra.run.smithi199.stdout:1 2019-12-11T16:32:39.768 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new 0d433771-717a-48b9-921d-2c9fe8e436b0 2 2019-12-11T16:32:41.036 INFO:teuthology.orchestra.run.smithi199.stdout:2 2019-12-11T16:32:41.060 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new 82673036-b875-46ee-87c1-4543fadf7518 3 2019-12-11T16:32:42.161 INFO:teuthology.orchestra.run.smithi199.stdout:3 2019-12-11T16:32:42.161 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new de52bcd2-0775-47f0-9e24-f64b800adf90 4 2019-12-11T16:32:43.284 INFO:teuthology.orchestra.run.smithi199.stdout:4 2019-12-11T16:32:43.311 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new 39d1b071-834e-4169-8ac3-883290f7728c 5 2019-12-11T16:32:44.304 INFO:teuthology.orchestra.run.smithi199.stdout:5 2019-12-11T16:32:44.337 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new 83ddf9b3-0c83-4ca7-a25a-175c3fcf1625 6 2019-12-11T16:32:45.449 INFO:teuthology.orchestra.run.smithi199.stdout:6 2019-12-11T16:32:45.468 INFO:teuthology.orchestra.run.smithi199:> sudo ceph --cluster ceph osd new 46354a68-efd7-47a7-ab2d-f49581a75936 7 2019-12-11T16:32:46.555 INFO:teuthology.orchestra.run.smithi199.stdout:7 2019-12-11T16:32:46.577 DEBUG:teuthology.misc:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.577 INFO:tasks.ceph.osd.0:Restarting daemon 2019-12-11T16:32:46.577 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 0 2019-12-11T16:32:46.581 INFO:tasks.ceph.osd.0:Started 2019-12-11T16:32:46.581 DEBUG:teuthology.misc:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.581 INFO:tasks.ceph.osd.1:Restarting daemon 2019-12-11T16:32:46.581 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 1 2019-12-11T16:32:46.584 INFO:tasks.ceph.osd.1:Started 2019-12-11T16:32:46.585 DEBUG:teuthology.misc:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.585 INFO:tasks.ceph.osd.2:Restarting daemon 2019-12-11T16:32:46.585 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 2 2019-12-11T16:32:46.588 INFO:tasks.ceph.osd.2:Started 2019-12-11T16:32:46.588 DEBUG:teuthology.misc:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.588 INFO:tasks.ceph.osd.3:Restarting daemon 2019-12-11T16:32:46.588 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 3 2019-12-11T16:32:46.591 INFO:tasks.ceph.osd.3:Started 2019-12-11T16:32:46.591 DEBUG:teuthology.misc:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.591 INFO:tasks.ceph.osd.4:Restarting daemon 2019-12-11T16:32:46.591 INFO:teuthology.orchestra.run.smithi199:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 4 2019-12-11T16:32:46.595 INFO:tasks.ceph.osd.4:Started 2019-12-11T16:32:46.595 DEBUG:teuthology.misc:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.595 INFO:tasks.ceph.osd.5:Restarting daemon 2019-12-11T16:32:46.595 INFO:teuthology.orchestra.run.smithi199:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 5 2019-12-11T16:32:46.598 INFO:tasks.ceph.osd.5:Started 2019-12-11T16:32:46.598 DEBUG:teuthology.misc:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.598 INFO:tasks.ceph.osd.6:Restarting daemon 2019-12-11T16:32:46.599 INFO:teuthology.orchestra.run.smithi199:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 6 2019-12-11T16:32:46.607 INFO:tasks.ceph.osd.6:Started 2019-12-11T16:32:46.608 DEBUG:teuthology.misc:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-11T16:32:46.608 INFO:tasks.ceph.osd.7:Restarting daemon 2019-12-11T16:32:46.608 INFO:teuthology.orchestra.run.smithi199:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 7 2019-12-11T16:32:46.611 INFO:tasks.ceph.osd.7:Started 2019-12-11T16:32:46.612 INFO:tasks.ceph:Waiting for OSDs to come up 2019-12-11T16:32:47.421 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-11T16:32:48.287 INFO:teuthology.misc.health.smithi079.stdout: 2019-12-11T16:32:48.287 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"b3142dac-fc19-47e9-ad13-f81d4b6261ad","created":"2019-12-11T16:32:31.386932+0000","modified":"2019-12-11T16:32:46.421818+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"1a8edef5-7984-4751-b15c-7c7aac7d8ca3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a652677c-9e77-4b93-b8d5-16a6b74760ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0d433771-717a-48b9-921d-2c9fe8e436b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82673036-b875-46ee-87c1-4543fadf7518","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"de52bcd2-0775-47f0-9e24-f64b800adf90","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"39d1b071-834e-4169-8ac3-883290f7728c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"83ddf9b3-0c83-4ca7-a25a-175c3fcf1625","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46354a68-efd7-47a7-ab2d-f49581a75936","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-11T16:32:48.298 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-12-11T16:32:50.337 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:03.600 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.359 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:03.619 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.413 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:03.669 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.414 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:03.678 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.416 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:03.676 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.496 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:03.752 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.502 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:03.763 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.563 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:03.824 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.669 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:03.941 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.725 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:04.000 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.806 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:04.131 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:50.865 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:04.192 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.014 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T16:32:50.993+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.024 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:04.298 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.032 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T16:32:51.007+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.082 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:04.356 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.100 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T16:32:51.084+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.117 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:04.442 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.162 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T16:32:51.150+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.171 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:04.499 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.284 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:04.548 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.287 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:04.547 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.308 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:04.565 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.336 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T16:32:51.312+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.369 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:04.630 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.470 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T16:32:51.451+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.532 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:04.808 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.667 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:04.927 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.679 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:04.938 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.694 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:04.958 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.708 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:04.972 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.712 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:04.988 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.716 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:04.972 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.724 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T16:32:51.706+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.728 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:05.003 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.736 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:05.065 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.742 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:04.999 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.792 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T16:32:51.780+0000 9b13800 -1 Falling back to public interface 2019-12-11T16:32:51.812 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:05.072 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.830 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:05.090 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:51.941 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:05.206 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:52.015 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:05.342 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:52.128 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:05.459 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:52.143 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:05.474 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:52.361 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:05.631 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:52.379 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:05.650 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:52.429 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:05.752 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:52.444 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:05.770 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:54.877 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:08.149 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:54.883 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:08.158 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.122 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-11T16:32:55.147 INFO:tasks.ceph.osd.3.smithi079.stderr:==00:00:00:08.423 10236== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.169 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:08.428 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.180 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:08.439 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.277 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:08.537 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.285 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:08.545 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.349 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:08.677 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.355 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:08.682 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.446 INFO:tasks.ceph.osd.4.smithi199.stderr:==00:00:00:08.706 10150== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.547 INFO:tasks.ceph.osd.6.smithi199.stderr:==00:00:00:08.811 10148== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.600 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:08.874 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.605 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:08.879 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.618 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:08.945 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.621 INFO:tasks.ceph.osd.0.smithi079.stderr:==00:00:00:08.946 10221== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.621 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:08.951 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.793 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:09.049 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.803 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:09.059 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.833 INFO:teuthology.misc.health.smithi079.stdout: 2019-12-11T16:32:55.833 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"b3142dac-fc19-47e9-ad13-f81d4b6261ad","created":"2019-12-11T16:32:31.386932+0000","modified":"2019-12-11T16:32:46.421818+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"1a8edef5-7984-4751-b15c-7c7aac7d8ca3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a652677c-9e77-4b93-b8d5-16a6b74760ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0d433771-717a-48b9-921d-2c9fe8e436b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82673036-b875-46ee-87c1-4543fadf7518","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"de52bcd2-0775-47f0-9e24-f64b800adf90","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"39d1b071-834e-4169-8ac3-883290f7728c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"83ddf9b3-0c83-4ca7-a25a-175c3fcf1625","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46354a68-efd7-47a7-ab2d-f49581a75936","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-11T16:32:55.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-12-11T16:32:55.867 INFO:tasks.ceph.osd.2.smithi079.stderr:==00:00:00:09.141 10237== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.883 INFO:tasks.ceph.osd.1.smithi079.stderr:==00:00:00:09.214 10214== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.989 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:09.250 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:55.997 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:09.258 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:56.067 INFO:tasks.ceph.osd.7.smithi199.stderr:==00:00:00:09.323 10151== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:56.258 INFO:tasks.ceph.osd.5.smithi199.stderr:==00:00:00:09.519 10149== Warning: unimplemented fcntl command: 1038 2019-12-11T16:32:56.769 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T16:32:56.770+0000 9b13800 -1 osd.3 0 log_to_monitors {default=true} 2019-12-11T16:32:56.993 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T16:32:56.987+0000 9b13800 -1 osd.4 0 log_to_monitors {default=true} 2019-12-11T16:32:57.062 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T16:32:57.063+0000 9b13800 -1 osd.0 0 log_to_monitors {default=true} 2019-12-11T16:32:57.120 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T16:32:57.117+0000 9b13800 -1 osd.6 0 log_to_monitors {default=true} 2019-12-11T16:32:57.249 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T16:32:57.243+0000 9b13800 -1 osd.2 0 log_to_monitors {default=true} 2019-12-11T16:32:57.355 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T16:32:57.355+0000 9b13800 -1 osd.1 0 log_to_monitors {default=true} 2019-12-11T16:32:57.454 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T16:32:57.453+0000 9b13800 -1 osd.7 0 log_to_monitors {default=true} 2019-12-11T16:32:57.749 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T16:32:57.737+0000 9b13800 -1 osd.5 0 log_to_monitors {default=true} 2019-12-11T16:32:59.080 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T16:32:59.082+0000 27772700 -1 osd.0 0 waiting for initial osdmap 2019-12-11T16:32:59.149 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T16:32:59.147+0000 2476e700 -1 osd.1 0 waiting for initial osdmap 2019-12-11T16:32:59.157 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T16:32:59.150+0000 27772700 -1 osd.2 0 waiting for initial osdmap 2019-12-11T16:32:59.176 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T16:32:59.178+0000 27772700 -1 osd.3 0 waiting for initial osdmap 2019-12-11T16:32:59.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T16:32:59.240+0000 2476e700 -1 osd.6 0 waiting for initial osdmap 2019-12-11T16:32:59.281 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T16:32:59.266+0000 27772700 -1 osd.4 0 waiting for initial osdmap 2019-12-11T16:33:00.378 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T16:33:00.375+0000 2476e700 -1 osd.5 0 waiting for initial osdmap 2019-12-11T16:33:00.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T16:33:00.385+0000 27b72700 -1 osd.7 0 waiting for initial osdmap 2019-12-11T16:33:02.655 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-11T16:33:03.506 INFO:teuthology.misc.health.smithi079.stdout: 2019-12-11T16:33:03.507 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":15,"fsid":"b3142dac-fc19-47e9-ad13-f81d4b6261ad","created":"2019-12-11T16:32:31.386932+0000","modified":"2019-12-11T16:33:02.052539+0000","last_up_change":"2019-12-11T16:33:02.052539+0000","last_in_change":"2019-12-11T16:33:02.052539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"1a8edef5-7984-4751-b15c-7c7aac7d8ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":10221},{"type":"v1","addr":"172.21.15.79:6827","nonce":10221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":10221},{"type":"v1","addr":"172.21.15.79:6829","nonce":10221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":10221},{"type":"v1","addr":"172.21.15.79:6833","nonce":10221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":10221},{"type":"v1","addr":"172.21.15.79:6831","nonce":10221}]},"public_addr":"172.21.15.79:6827/10221","cluster_addr":"172.21.15.79:6829/10221","heartbeat_back_addr":"172.21.15.79:6833/10221","heartbeat_front_addr":"172.21.15.79:6831/10221","state":["exists","up"]},{"osd":1,"uuid":"a652677c-9e77-4b93-b8d5-16a6b74760ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":10214},{"type":"v1","addr":"172.21.15.79:6811","nonce":10214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":10214},{"type":"v1","addr":"172.21.15.79:6813","nonce":10214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":10214},{"type":"v1","addr":"172.21.15.79:6817","nonce":10214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":10214},{"type":"v1","addr":"172.21.15.79:6815","nonce":10214}]},"public_addr":"172.21.15.79:6811/10214","cluster_addr":"172.21.15.79:6813/10214","heartbeat_back_addr":"172.21.15.79:6817/10214","heartbeat_front_addr":"172.21.15.79:6815/10214","state":["exists","up"]},{"osd":2,"uuid":"0d433771-717a-48b9-921d-2c9fe8e436b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":10237},{"type":"v1","addr":"172.21.15.79:6819","nonce":10237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":10237},{"type":"v1","addr":"172.21.15.79:6821","nonce":10237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":10237},{"type":"v1","addr":"172.21.15.79:6825","nonce":10237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":10237},{"type":"v1","addr":"172.21.15.79:6823","nonce":10237}]},"public_addr":"172.21.15.79:6819/10237","cluster_addr":"172.21.15.79:6821/10237","heartbeat_back_addr":"172.21.15.79:6825/10237","heartbeat_front_addr":"172.21.15.79:6823/10237","state":["exists","up"]},{"osd":3,"uuid":"82673036-b875-46ee-87c1-4543fadf7518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":10236},{"type":"v1","addr":"172.21.15.79:6803","nonce":10236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":10236},{"type":"v1","addr":"172.21.15.79:6805","nonce":10236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":10236},{"type":"v1","addr":"172.21.15.79:6809","nonce":10236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":10236},{"type":"v1","addr":"172.21.15.79:6807","nonce":10236}]},"public_addr":"172.21.15.79:6803/10236","cluster_addr":"172.21.15.79:6805/10236","heartbeat_back_addr":"172.21.15.79:6809/10236","heartbeat_front_addr":"172.21.15.79:6807/10236","state":["exists","up"]},{"osd":4,"uuid":"de52bcd2-0775-47f0-9e24-f64b800adf90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6806","nonce":10150},{"type":"v1","addr":"172.21.15.199:6809","nonce":10150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6810","nonce":10150},{"type":"v1","addr":"172.21.15.199:6811","nonce":10150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6814","nonce":10150},{"type":"v1","addr":"172.21.15.199:6815","nonce":10150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6812","nonce":10150},{"type":"v1","addr":"172.21.15.199:6813","nonce":10150}]},"public_addr":"172.21.15.199:6809/10150","cluster_addr":"172.21.15.199:6811/10150","heartbeat_back_addr":"172.21.15.199:6815/10150","heartbeat_front_addr":"172.21.15.199:6813/10150","state":["exists","up"]},{"osd":5,"uuid":"39d1b071-834e-4169-8ac3-883290f7728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6824","nonce":10149},{"type":"v1","addr":"172.21.15.199:6825","nonce":10149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6826","nonce":10149},{"type":"v1","addr":"172.21.15.199:6827","nonce":10149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6830","nonce":10149},{"type":"v1","addr":"172.21.15.199:6831","nonce":10149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6828","nonce":10149},{"type":"v1","addr":"172.21.15.199:6829","nonce":10149}]},"public_addr":"172.21.15.199:6825/10149","cluster_addr":"172.21.15.199:6827/10149","heartbeat_back_addr":"172.21.15.199:6831/10149","heartbeat_front_addr":"172.21.15.199:6829/10149","state":["exists","up"]},{"osd":6,"uuid":"83ddf9b3-0c83-4ca7-a25a-175c3fcf1625","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6800","nonce":10148},{"type":"v1","addr":"172.21.15.199:6801","nonce":10148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6802","nonce":10148},{"type":"v1","addr":"172.21.15.199:6803","nonce":10148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6807","nonce":10148},{"type":"v1","addr":"172.21.15.199:6808","nonce":10148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6804","nonce":10148},{"type":"v1","addr":"172.21.15.199:6805","nonce":10148}]},"public_addr":"172.21.15.199:6801/10148","cluster_addr":"172.21.15.199:6803/10148","heartbeat_back_addr":"172.21.15.199:6808/10148","heartbeat_front_addr":"172.21.15.199:6805/10148","state":["exists","up"]},{"osd":7,"uuid":"46354a68-efd7-47a7-ab2d-f49581a75936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6816","nonce":10151},{"type":"v1","addr":"172.21.15.199:6817","nonce":10151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6818","nonce":10151},{"type":"v1","addr":"172.21.15.199:6819","nonce":10151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6822","nonce":10151},{"type":"v1","addr":"172.21.15.199:6823","nonce":10151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6820","nonce":10151},{"type":"v1","addr":"172.21.15.199:6821","nonce":10151}]},"public_addr":"172.21.15.199:6817/10151","cluster_addr":"172.21.15.199:6819/10151","heartbeat_back_addr":"172.21.15.199:6823/10151","heartbeat_front_addr":"172.21.15.199:6821/10151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.188983+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.430567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.396462+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:57.857492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.236957+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-11T16:33:03.524 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-12-11T16:33:03.524 INFO:tasks.ceph:Creating RBD pool 2019-12-11T16:33:03.524 INFO:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool create rbd 8 2019-12-11T16:33:05.756 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'rbd' created 2019-12-11T16:33:05.776 INFO:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-12-11T16:33:07.862 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rbd' on pool 'rbd' 2019-12-11T16:33:07.876 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-12-11T16:33:07.876 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-11T16:33:07.877 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-12-11T16:33:08.525 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:33:08.525 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":19,"fsid":"b3142dac-fc19-47e9-ad13-f81d4b6261ad","created":"2019-12-11T16:32:31.386932+0000","modified":"2019-12-11T16:33:07.561797+0000","last_up_change":"2019-12-11T16:33:02.052539+0000","last_in_change":"2019-12-11T16:33:02.052539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-11T16:33:04.421535+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1a8edef5-7984-4751-b15c-7c7aac7d8ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":10221},{"type":"v1","addr":"172.21.15.79:6827","nonce":10221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":10221},{"type":"v1","addr":"172.21.15.79:6829","nonce":10221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":10221},{"type":"v1","addr":"172.21.15.79:6833","nonce":10221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":10221},{"type":"v1","addr":"172.21.15.79:6831","nonce":10221}]},"public_addr":"172.21.15.79:6827/10221","cluster_addr":"172.21.15.79:6829/10221","heartbeat_back_addr":"172.21.15.79:6833/10221","heartbeat_front_addr":"172.21.15.79:6831/10221","state":["exists","up"]},{"osd":1,"uuid":"a652677c-9e77-4b93-b8d5-16a6b74760ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":10214},{"type":"v1","addr":"172.21.15.79:6811","nonce":10214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":10214},{"type":"v1","addr":"172.21.15.79:6813","nonce":10214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":10214},{"type":"v1","addr":"172.21.15.79:6817","nonce":10214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":10214},{"type":"v1","addr":"172.21.15.79:6815","nonce":10214}]},"public_addr":"172.21.15.79:6811/10214","cluster_addr":"172.21.15.79:6813/10214","heartbeat_back_addr":"172.21.15.79:6817/10214","heartbeat_front_addr":"172.21.15.79:6815/10214","state":["exists","up"]},{"osd":2,"uuid":"0d433771-717a-48b9-921d-2c9fe8e436b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":10237},{"type":"v1","addr":"172.21.15.79:6819","nonce":10237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":10237},{"type":"v1","addr":"172.21.15.79:6821","nonce":10237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":10237},{"type":"v1","addr":"172.21.15.79:6825","nonce":10237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":10237},{"type":"v1","addr":"172.21.15.79:6823","nonce":10237}]},"public_addr":"172.21.15.79:6819/10237","cluster_addr":"172.21.15.79:6821/10237","heartbeat_back_addr":"172.21.15.79:6825/10237","heartbeat_front_addr":"172.21.15.79:6823/10237","state":["exists","up"]},{"osd":3,"uuid":"82673036-b875-46ee-87c1-4543fadf7518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":10236},{"type":"v1","addr":"172.21.15.79:6803","nonce":10236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":10236},{"type":"v1","addr":"172.21.15.79:6805","nonce":10236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":10236},{"type":"v1","addr":"172.21.15.79:6809","nonce":10236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":10236},{"type":"v1","addr":"172.21.15.79:6807","nonce":10236}]},"public_addr":"172.21.15.79:6803/10236","cluster_addr":"172.21.15.79:6805/10236","heartbeat_back_addr":"172.21.15.79:6809/10236","heartbeat_front_addr":"172.21.15.79:6807/10236","state":["exists","up"]},{"osd":4,"uuid":"de52bcd2-0775-47f0-9e24-f64b800adf90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6806","nonce":10150},{"type":"v1","addr":"172.21.15.199:6809","nonce":10150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6810","nonce":10150},{"type":"v1","addr":"172.21.15.199:6811","nonce":10150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6814","nonce":10150},{"type":"v1","addr":"172.21.15.199:6815","nonce":10150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6812","nonce":10150},{"type":"v1","addr":"172.21.15.199:6813","nonce":10150}]},"public_addr":"172.21.15.199:6809/10150","cluster_addr":"172.21.15.199:6811/10150","heartbeat_back_addr":"172.21.15.199:6815/10150","heartbeat_front_addr":"172.21.15.199:6813/10150","state":["exists","up"]},{"osd":5,"uuid":"39d1b071-834e-4169-8ac3-883290f7728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6824","nonce":10149},{"type":"v1","addr":"172.21.15.199:6825","nonce":10149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6826","nonce":10149},{"type":"v1","addr":"172.21.15.199:6827","nonce":10149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6830","nonce":10149},{"type":"v1","addr":"172.21.15.199:6831","nonce":10149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6828","nonce":10149},{"type":"v1","addr":"172.21.15.199:6829","nonce":10149}]},"public_addr":"172.21.15.199:6825/10149","cluster_addr":"172.21.15.199:6827/10149","heartbeat_back_addr":"172.21.15.199:6831/10149","heartbeat_front_addr":"172.21.15.199:6829/10149","state":["exists","up"]},{"osd":6,"uuid":"83ddf9b3-0c83-4ca7-a25a-175c3fcf1625","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6800","nonce":10148},{"type":"v1","addr":"172.21.15.199:6801","nonce":10148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6802","nonce":10148},{"type":"v1","addr":"172.21.15.199:6803","nonce":10148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6807","nonce":10148},{"type":"v1","addr":"172.21.15.199:6808","nonce":10148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6804","nonce":10148},{"type":"v1","addr":"172.21.15.199:6805","nonce":10148}]},"public_addr":"172.21.15.199:6801/10148","cluster_addr":"172.21.15.199:6803/10148","heartbeat_back_addr":"172.21.15.199:6808/10148","heartbeat_front_addr":"172.21.15.199:6805/10148","state":["exists","up"]},{"osd":7,"uuid":"46354a68-efd7-47a7-ab2d-f49581a75936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6816","nonce":10151},{"type":"v1","addr":"172.21.15.199:6817","nonce":10151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6818","nonce":10151},{"type":"v1","addr":"172.21.15.199:6819","nonce":10151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6822","nonce":10151},{"type":"v1","addr":"172.21.15.199:6823","nonce":10151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6820","nonce":10151},{"type":"v1","addr":"172.21.15.199:6821","nonce":10151}]},"public_addr":"172.21.15.199:6817/10151","cluster_addr":"172.21.15.199:6819/10151","heartbeat_back_addr":"172.21.15.199:6823/10151","heartbeat_front_addr":"172.21.15.199:6821/10151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.188983+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.430567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.396462+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:57.857492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.137355+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.856964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.236957+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.614138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-11T16:33:08.538 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-11T16:33:04.421535+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'19', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-11T16:33:08.538 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-11T16:33:09.489 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 8 2019-12-11T16:33:09.506 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-12-11T16:33:09.506 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-12-11T16:33:09.506 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-12-11T16:33:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:33:10.498 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":3,"active_gid":4108,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":9284},{"type":"v1","addr":"172.21.15.79:6801","nonce":9284}]},"active_addr":"172.21.15.79:6801/9284","active_change":"2019-12-11T16:32:35.129416+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]}} 2019-12-11T16:33:10.526 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-12-11T16:33:10.526 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2019-12-11T16:33:10.526 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-11T16:33:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:33:11.308 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":20,"fsid":"b3142dac-fc19-47e9-ad13-f81d4b6261ad","created":"2019-12-11T16:32:31.386932+0000","modified":"2019-12-11T16:33:09.291712+0000","last_up_change":"2019-12-11T16:33:02.052539+0000","last_in_change":"2019-12-11T16:33:02.052539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-11T16:33:04.421535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1a8edef5-7984-4751-b15c-7c7aac7d8ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":10221},{"type":"v1","addr":"172.21.15.79:6827","nonce":10221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":10221},{"type":"v1","addr":"172.21.15.79:6829","nonce":10221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":10221},{"type":"v1","addr":"172.21.15.79:6833","nonce":10221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":10221},{"type":"v1","addr":"172.21.15.79:6831","nonce":10221}]},"public_addr":"172.21.15.79:6827/10221","cluster_addr":"172.21.15.79:6829/10221","heartbeat_back_addr":"172.21.15.79:6833/10221","heartbeat_front_addr":"172.21.15.79:6831/10221","state":["exists","up"]},{"osd":1,"uuid":"a652677c-9e77-4b93-b8d5-16a6b74760ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":10214},{"type":"v1","addr":"172.21.15.79:6811","nonce":10214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":10214},{"type":"v1","addr":"172.21.15.79:6813","nonce":10214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":10214},{"type":"v1","addr":"172.21.15.79:6817","nonce":10214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":10214},{"type":"v1","addr":"172.21.15.79:6815","nonce":10214}]},"public_addr":"172.21.15.79:6811/10214","cluster_addr":"172.21.15.79:6813/10214","heartbeat_back_addr":"172.21.15.79:6817/10214","heartbeat_front_addr":"172.21.15.79:6815/10214","state":["exists","up"]},{"osd":2,"uuid":"0d433771-717a-48b9-921d-2c9fe8e436b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":10237},{"type":"v1","addr":"172.21.15.79:6819","nonce":10237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":10237},{"type":"v1","addr":"172.21.15.79:6821","nonce":10237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":10237},{"type":"v1","addr":"172.21.15.79:6825","nonce":10237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":10237},{"type":"v1","addr":"172.21.15.79:6823","nonce":10237}]},"public_addr":"172.21.15.79:6819/10237","cluster_addr":"172.21.15.79:6821/10237","heartbeat_back_addr":"172.21.15.79:6825/10237","heartbeat_front_addr":"172.21.15.79:6823/10237","state":["exists","up"]},{"osd":3,"uuid":"82673036-b875-46ee-87c1-4543fadf7518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":10236},{"type":"v1","addr":"172.21.15.79:6803","nonce":10236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":10236},{"type":"v1","addr":"172.21.15.79:6805","nonce":10236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":10236},{"type":"v1","addr":"172.21.15.79:6809","nonce":10236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":10236},{"type":"v1","addr":"172.21.15.79:6807","nonce":10236}]},"public_addr":"172.21.15.79:6803/10236","cluster_addr":"172.21.15.79:6805/10236","heartbeat_back_addr":"172.21.15.79:6809/10236","heartbeat_front_addr":"172.21.15.79:6807/10236","state":["exists","up"]},{"osd":4,"uuid":"de52bcd2-0775-47f0-9e24-f64b800adf90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6806","nonce":10150},{"type":"v1","addr":"172.21.15.199:6809","nonce":10150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6810","nonce":10150},{"type":"v1","addr":"172.21.15.199:6811","nonce":10150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6814","nonce":10150},{"type":"v1","addr":"172.21.15.199:6815","nonce":10150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6812","nonce":10150},{"type":"v1","addr":"172.21.15.199:6813","nonce":10150}]},"public_addr":"172.21.15.199:6809/10150","cluster_addr":"172.21.15.199:6811/10150","heartbeat_back_addr":"172.21.15.199:6815/10150","heartbeat_front_addr":"172.21.15.199:6813/10150","state":["exists","up"]},{"osd":5,"uuid":"39d1b071-834e-4169-8ac3-883290f7728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6824","nonce":10149},{"type":"v1","addr":"172.21.15.199:6825","nonce":10149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6826","nonce":10149},{"type":"v1","addr":"172.21.15.199:6827","nonce":10149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6830","nonce":10149},{"type":"v1","addr":"172.21.15.199:6831","nonce":10149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6828","nonce":10149},{"type":"v1","addr":"172.21.15.199:6829","nonce":10149}]},"public_addr":"172.21.15.199:6825/10149","cluster_addr":"172.21.15.199:6827/10149","heartbeat_back_addr":"172.21.15.199:6831/10149","heartbeat_front_addr":"172.21.15.199:6829/10149","state":["exists","up"]},{"osd":6,"uuid":"83ddf9b3-0c83-4ca7-a25a-175c3fcf1625","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6800","nonce":10148},{"type":"v1","addr":"172.21.15.199:6801","nonce":10148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6802","nonce":10148},{"type":"v1","addr":"172.21.15.199:6803","nonce":10148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6807","nonce":10148},{"type":"v1","addr":"172.21.15.199:6808","nonce":10148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6804","nonce":10148},{"type":"v1","addr":"172.21.15.199:6805","nonce":10148}]},"public_addr":"172.21.15.199:6801/10148","cluster_addr":"172.21.15.199:6803/10148","heartbeat_back_addr":"172.21.15.199:6808/10148","heartbeat_front_addr":"172.21.15.199:6805/10148","state":["exists","up"]},{"osd":7,"uuid":"46354a68-efd7-47a7-ab2d-f49581a75936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6816","nonce":10151},{"type":"v1","addr":"172.21.15.199:6817","nonce":10151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6818","nonce":10151},{"type":"v1","addr":"172.21.15.199:6819","nonce":10151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6822","nonce":10151},{"type":"v1","addr":"172.21.15.199:6823","nonce":10151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6820","nonce":10151},{"type":"v1","addr":"172.21.15.199:6821","nonce":10151}]},"public_addr":"172.21.15.199:6817/10151","cluster_addr":"172.21.15.199:6819/10151","heartbeat_back_addr":"172.21.15.199:6823/10151","heartbeat_front_addr":"172.21.15.199:6821/10151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.188983+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.430567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.396462+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:57.857492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.137355+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.856964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.236957+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.614138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-11T16:33:11.322 INFO:tasks.ceph.ceph_manager.ceph:all up! 2019-12-11T16:33:11.323 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-11T16:33:12.101 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:33:12.102 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":20,"fsid":"b3142dac-fc19-47e9-ad13-f81d4b6261ad","created":"2019-12-11T16:32:31.386932+0000","modified":"2019-12-11T16:33:09.291712+0000","last_up_change":"2019-12-11T16:33:02.052539+0000","last_in_change":"2019-12-11T16:33:02.052539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-11T16:33:04.421535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1a8edef5-7984-4751-b15c-7c7aac7d8ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":10221},{"type":"v1","addr":"172.21.15.79:6827","nonce":10221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":10221},{"type":"v1","addr":"172.21.15.79:6829","nonce":10221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":10221},{"type":"v1","addr":"172.21.15.79:6833","nonce":10221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":10221},{"type":"v1","addr":"172.21.15.79:6831","nonce":10221}]},"public_addr":"172.21.15.79:6827/10221","cluster_addr":"172.21.15.79:6829/10221","heartbeat_back_addr":"172.21.15.79:6833/10221","heartbeat_front_addr":"172.21.15.79:6831/10221","state":["exists","up"]},{"osd":1,"uuid":"a652677c-9e77-4b93-b8d5-16a6b74760ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":10214},{"type":"v1","addr":"172.21.15.79:6811","nonce":10214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":10214},{"type":"v1","addr":"172.21.15.79:6813","nonce":10214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":10214},{"type":"v1","addr":"172.21.15.79:6817","nonce":10214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":10214},{"type":"v1","addr":"172.21.15.79:6815","nonce":10214}]},"public_addr":"172.21.15.79:6811/10214","cluster_addr":"172.21.15.79:6813/10214","heartbeat_back_addr":"172.21.15.79:6817/10214","heartbeat_front_addr":"172.21.15.79:6815/10214","state":["exists","up"]},{"osd":2,"uuid":"0d433771-717a-48b9-921d-2c9fe8e436b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":10237},{"type":"v1","addr":"172.21.15.79:6819","nonce":10237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":10237},{"type":"v1","addr":"172.21.15.79:6821","nonce":10237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":10237},{"type":"v1","addr":"172.21.15.79:6825","nonce":10237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":10237},{"type":"v1","addr":"172.21.15.79:6823","nonce":10237}]},"public_addr":"172.21.15.79:6819/10237","cluster_addr":"172.21.15.79:6821/10237","heartbeat_back_addr":"172.21.15.79:6825/10237","heartbeat_front_addr":"172.21.15.79:6823/10237","state":["exists","up"]},{"osd":3,"uuid":"82673036-b875-46ee-87c1-4543fadf7518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":10236},{"type":"v1","addr":"172.21.15.79:6803","nonce":10236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":10236},{"type":"v1","addr":"172.21.15.79:6805","nonce":10236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":10236},{"type":"v1","addr":"172.21.15.79:6809","nonce":10236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":10236},{"type":"v1","addr":"172.21.15.79:6807","nonce":10236}]},"public_addr":"172.21.15.79:6803/10236","cluster_addr":"172.21.15.79:6805/10236","heartbeat_back_addr":"172.21.15.79:6809/10236","heartbeat_front_addr":"172.21.15.79:6807/10236","state":["exists","up"]},{"osd":4,"uuid":"de52bcd2-0775-47f0-9e24-f64b800adf90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6806","nonce":10150},{"type":"v1","addr":"172.21.15.199:6809","nonce":10150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6810","nonce":10150},{"type":"v1","addr":"172.21.15.199:6811","nonce":10150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6814","nonce":10150},{"type":"v1","addr":"172.21.15.199:6815","nonce":10150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6812","nonce":10150},{"type":"v1","addr":"172.21.15.199:6813","nonce":10150}]},"public_addr":"172.21.15.199:6809/10150","cluster_addr":"172.21.15.199:6811/10150","heartbeat_back_addr":"172.21.15.199:6815/10150","heartbeat_front_addr":"172.21.15.199:6813/10150","state":["exists","up"]},{"osd":5,"uuid":"39d1b071-834e-4169-8ac3-883290f7728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6824","nonce":10149},{"type":"v1","addr":"172.21.15.199:6825","nonce":10149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6826","nonce":10149},{"type":"v1","addr":"172.21.15.199:6827","nonce":10149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6830","nonce":10149},{"type":"v1","addr":"172.21.15.199:6831","nonce":10149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6828","nonce":10149},{"type":"v1","addr":"172.21.15.199:6829","nonce":10149}]},"public_addr":"172.21.15.199:6825/10149","cluster_addr":"172.21.15.199:6827/10149","heartbeat_back_addr":"172.21.15.199:6831/10149","heartbeat_front_addr":"172.21.15.199:6829/10149","state":["exists","up"]},{"osd":6,"uuid":"83ddf9b3-0c83-4ca7-a25a-175c3fcf1625","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6800","nonce":10148},{"type":"v1","addr":"172.21.15.199:6801","nonce":10148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6802","nonce":10148},{"type":"v1","addr":"172.21.15.199:6803","nonce":10148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6807","nonce":10148},{"type":"v1","addr":"172.21.15.199:6808","nonce":10148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6804","nonce":10148},{"type":"v1","addr":"172.21.15.199:6805","nonce":10148}]},"public_addr":"172.21.15.199:6801/10148","cluster_addr":"172.21.15.199:6803/10148","heartbeat_back_addr":"172.21.15.199:6808/10148","heartbeat_front_addr":"172.21.15.199:6805/10148","state":["exists","up"]},{"osd":7,"uuid":"46354a68-efd7-47a7-ab2d-f49581a75936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6816","nonce":10151},{"type":"v1","addr":"172.21.15.199:6817","nonce":10151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6818","nonce":10151},{"type":"v1","addr":"172.21.15.199:6819","nonce":10151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6822","nonce":10151},{"type":"v1","addr":"172.21.15.199:6823","nonce":10151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.199:6820","nonce":10151},{"type":"v1","addr":"172.21.15.199:6821","nonce":10151}]},"public_addr":"172.21.15.199:6817/10151","cluster_addr":"172.21.15.199:6819/10151","heartbeat_back_addr":"172.21.15.199:6823/10151","heartbeat_front_addr":"172.21.15.199:6821/10151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.188983+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.430567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.396462+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:57.857492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.137355+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.856964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.236957+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-11T16:32:58.614138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-11T16:33:12.116 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-12-11T16:33:12.507 INFO:teuthology.orchestra.run.smithi079.stdout:55834574854 2019-12-11T16:33:12.507 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-12-11T16:33:12.980 INFO:teuthology.orchestra.run.smithi079.stdout:55834574854 2019-12-11T16:33:12.980 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-12-11T16:33:13.457 INFO:teuthology.orchestra.run.smithi079.stdout:55834574852 2019-12-11T16:33:13.458 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-12-11T16:33:13.917 INFO:teuthology.orchestra.run.smithi079.stdout:55834574853 2019-12-11T16:33:13.917 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-12-11T16:33:14.399 INFO:teuthology.orchestra.run.smithi079.stdout:60129542151 2019-12-11T16:33:14.400 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-12-11T16:33:14.943 INFO:teuthology.orchestra.run.smithi079.stdout:64424509445 2019-12-11T16:33:14.943 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-12-11T16:33:15.547 INFO:teuthology.orchestra.run.smithi079.stdout:60129542149 2019-12-11T16:33:15.548 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-12-11T16:33:16.158 INFO:teuthology.orchestra.run.smithi079.stdout:64424509446 2019-12-11T16:33:16.159 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-12-11T16:33:17.033 INFO:teuthology.orchestra.run.smithi079.stdout:55834574854 2019-12-11T16:33:17.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2019-12-11T16:33:17.048 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-12-11T16:33:17.720 INFO:teuthology.orchestra.run.smithi079.stdout:55834574854 2019-12-11T16:33:17.734 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2019-12-11T16:33:17.735 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-12-11T16:33:18.486 INFO:teuthology.orchestra.run.smithi079.stdout:55834574853 2019-12-11T16:33:18.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.2 2019-12-11T16:33:18.500 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-12-11T16:33:19.182 INFO:teuthology.orchestra.run.smithi079.stdout:55834574854 2019-12-11T16:33:19.196 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.3 2019-12-11T16:33:19.196 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-12-11T16:33:19.984 INFO:teuthology.orchestra.run.smithi079.stdout:60129542152 2019-12-11T16:33:19.999 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542152 for osd.4 2019-12-11T16:33:19.999 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-12-11T16:33:20.795 INFO:teuthology.orchestra.run.smithi079.stdout:64424509446 2019-12-11T16:33:20.809 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509446 for osd.5 2019-12-11T16:33:20.809 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-12-11T16:33:21.599 INFO:teuthology.orchestra.run.smithi079.stdout:60129542150 2019-12-11T16:33:21.613 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.6 2019-12-11T16:33:21.613 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-12-11T16:33:22.434 INFO:teuthology.orchestra.run.smithi079.stdout:64424509447 2019-12-11T16:33:22.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.7 2019-12-11T16:33:22.451 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-12-11T16:33:22.451 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-11T16:33:23.172 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2019-12-11T16:33:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:33:23.185 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2019-12-11T16:33:21.473874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390208,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584512,"statfs":{"total":773094113280,"available":764502540288,"internally_reserved":8589934592,"allocated":1572864,"data_stored":512736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:07.858790+0000","last_change":"2019-12-11T16:33:07.048407+0000","last_active":"2019-12-11T16:33:07.858790+0000","last_peered":"2019-12-11T16:33:07.858790+0000","last_clean":"2019-12-11T16:33:07.858790+0000","last_became_active":"2019-12-11T16:33:07.013768+0000","last_became_peered":"2019-12-11T16:33:07.013768+0000","last_unstale":"2019-12-11T16:33:07.858790+0000","last_undegraded":"2019-12-11T16:33:07.858790+0000","last_fullsized":"2019-12-11T16:33:07.858790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:07.848689+0000","last_change":"2019-12-11T16:33:07.152676+0000","last_active":"2019-12-11T16:33:07.848689+0000","last_peered":"2019-12-11T16:33:07.848689+0000","last_clean":"2019-12-11T16:33:07.848689+0000","last_became_active":"2019-12-11T16:33:07.136234+0000","last_became_peered":"2019-12-11T16:33:07.136234+0000","last_unstale":"2019-12-11T16:33:07.848689+0000","last_undegraded":"2019-12-11T16:33:07.848689+0000","last_fullsized":"2019-12-11T16:33:07.848689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:09.003493+0000","last_change":"2019-12-11T16:33:08.977401+0000","last_active":"2019-12-11T16:33:09.003493+0000","last_peered":"2019-12-11T16:33:09.003493+0000","last_clean":"2019-12-11T16:33:09.003493+0000","last_became_active":"2019-12-11T16:33:08.917007+0000","last_became_peered":"2019-12-11T16:33:08.917007+0000","last_unstale":"2019-12-11T16:33:09.003493+0000","last_undegraded":"2019-12-11T16:33:09.003493+0000","last_fullsized":"2019-12-11T16:33:09.003493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:07.833049+0000","last_change":"2019-12-11T16:33:06.955932+0000","last_active":"2019-12-11T16:33:07.833049+0000","last_peered":"2019-12-11T16:33:07.833049+0000","last_clean":"2019-12-11T16:33:07.833049+0000","last_became_active":"2019-12-11T16:33:06.885511+0000","last_became_peered":"2019-12-11T16:33:06.885511+0000","last_unstale":"2019-12-11T16:33:07.833049+0000","last_undegraded":"2019-12-11T16:33:07.833049+0000","last_fullsized":"2019-12-11T16:33:07.833049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:08.653788+0000","last_change":"2019-12-11T16:33:08.594542+0000","last_active":"2019-12-11T16:33:08.653788+0000","last_peered":"2019-12-11T16:33:08.653788+0000","last_clean":"2019-12-11T16:33:08.653788+0000","last_became_active":"2019-12-11T16:33:08.530961+0000","last_became_peered":"2019-12-11T16:33:08.530961+0000","last_unstale":"2019-12-11T16:33:08.653788+0000","last_undegraded":"2019-12-11T16:33:08.653788+0000","last_fullsized":"2019-12-11T16:33:08.653788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:08.569617+0000","last_change":"2019-12-11T16:33:07.173960+0000","last_active":"2019-12-11T16:33:08.569617+0000","last_peered":"2019-12-11T16:33:08.569617+0000","last_clean":"2019-12-11T16:33:08.569617+0000","last_became_active":"2019-12-11T16:33:07.143462+0000","last_became_peered":"2019-12-11T16:33:07.143462+0000","last_unstale":"2019-12-11T16:33:08.569617+0000","last_undegraded":"2019-12-11T16:33:08.569617+0000","last_fullsized":"2019-12-11T16:33:08.569617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:08.659022+0000","last_change":"2019-12-11T16:33:08.637025+0000","last_active":"2019-12-11T16:33:08.659022+0000","last_peered":"2019-12-11T16:33:08.659022+0000","last_clean":"2019-12-11T16:33:08.659022+0000","last_became_active":"2019-12-11T16:33:08.616039+0000","last_became_peered":"2019-12-11T16:33:08.616039+0000","last_unstale":"2019-12-11T16:33:08.659022+0000","last_undegraded":"2019-12-11T16:33:08.659022+0000","last_fullsized":"2019-12-11T16:33:08.659022+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:09.034637+0000","last_change":"2019-12-11T16:33:06.972480+0000","last_active":"2019-12-11T16:33:09.034637+0000","last_peered":"2019-12-11T16:33:09.034637+0000","last_clean":"2019-12-11T16:33:09.034637+0000","last_became_active":"2019-12-11T16:33:06.933094+0000","last_became_peered":"2019-12-11T16:33:06.933094+0000","last_unstale":"2019-12-11T16:33:09.034637+0000","last_undegraded":"2019-12-11T16:33:09.034637+0000","last_fullsized":"2019-12-11T16:33:09.034637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.771}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.000}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":35.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.659}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":27.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":40.080}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.520}]}]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.522}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.726}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.964}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":35.744}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":40.315}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":37.971}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":17.066}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.157}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.326}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":27.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.654}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":120.188}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":29.228}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.394}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.899}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.739}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.346}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.852}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.588}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.987}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":10.142}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":13.597}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":9.294}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":17.967}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.690}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.357}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":17.384}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.530}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.051}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.734}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.519}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.878}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.485}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":8.841}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":18.960}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.755}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.678}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.066}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":30.802}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.629}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":11.126}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":12.295}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":10.322}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":18.862}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.815}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.620}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.933}]}]}]}} 2019-12-11T16:33:23.187 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-11T16:33:23.875 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:33:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2019-12-11T16:33:23.888 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2019-12-11T16:33:23.474420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390208,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584512,"statfs":{"total":773094113280,"available":764502540288,"internally_reserved":8589934592,"allocated":1572864,"data_stored":512736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:07.858790+0000","last_change":"2019-12-11T16:33:07.048407+0000","last_active":"2019-12-11T16:33:07.858790+0000","last_peered":"2019-12-11T16:33:07.858790+0000","last_clean":"2019-12-11T16:33:07.858790+0000","last_became_active":"2019-12-11T16:33:07.013768+0000","last_became_peered":"2019-12-11T16:33:07.013768+0000","last_unstale":"2019-12-11T16:33:07.858790+0000","last_undegraded":"2019-12-11T16:33:07.858790+0000","last_fullsized":"2019-12-11T16:33:07.858790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:07.848689+0000","last_change":"2019-12-11T16:33:07.152676+0000","last_active":"2019-12-11T16:33:07.848689+0000","last_peered":"2019-12-11T16:33:07.848689+0000","last_clean":"2019-12-11T16:33:07.848689+0000","last_became_active":"2019-12-11T16:33:07.136234+0000","last_became_peered":"2019-12-11T16:33:07.136234+0000","last_unstale":"2019-12-11T16:33:07.848689+0000","last_undegraded":"2019-12-11T16:33:07.848689+0000","last_fullsized":"2019-12-11T16:33:07.848689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:09.003493+0000","last_change":"2019-12-11T16:33:08.977401+0000","last_active":"2019-12-11T16:33:09.003493+0000","last_peered":"2019-12-11T16:33:09.003493+0000","last_clean":"2019-12-11T16:33:09.003493+0000","last_became_active":"2019-12-11T16:33:08.917007+0000","last_became_peered":"2019-12-11T16:33:08.917007+0000","last_unstale":"2019-12-11T16:33:09.003493+0000","last_undegraded":"2019-12-11T16:33:09.003493+0000","last_fullsized":"2019-12-11T16:33:09.003493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:07.833049+0000","last_change":"2019-12-11T16:33:06.955932+0000","last_active":"2019-12-11T16:33:07.833049+0000","last_peered":"2019-12-11T16:33:07.833049+0000","last_clean":"2019-12-11T16:33:07.833049+0000","last_became_active":"2019-12-11T16:33:06.885511+0000","last_became_peered":"2019-12-11T16:33:06.885511+0000","last_unstale":"2019-12-11T16:33:07.833049+0000","last_undegraded":"2019-12-11T16:33:07.833049+0000","last_fullsized":"2019-12-11T16:33:07.833049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:08.653788+0000","last_change":"2019-12-11T16:33:08.594542+0000","last_active":"2019-12-11T16:33:08.653788+0000","last_peered":"2019-12-11T16:33:08.653788+0000","last_clean":"2019-12-11T16:33:08.653788+0000","last_became_active":"2019-12-11T16:33:08.530961+0000","last_became_peered":"2019-12-11T16:33:08.530961+0000","last_unstale":"2019-12-11T16:33:08.653788+0000","last_undegraded":"2019-12-11T16:33:08.653788+0000","last_fullsized":"2019-12-11T16:33:08.653788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:08.569617+0000","last_change":"2019-12-11T16:33:07.173960+0000","last_active":"2019-12-11T16:33:08.569617+0000","last_peered":"2019-12-11T16:33:08.569617+0000","last_clean":"2019-12-11T16:33:08.569617+0000","last_became_active":"2019-12-11T16:33:07.143462+0000","last_became_peered":"2019-12-11T16:33:07.143462+0000","last_unstale":"2019-12-11T16:33:08.569617+0000","last_undegraded":"2019-12-11T16:33:08.569617+0000","last_fullsized":"2019-12-11T16:33:08.569617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:08.659022+0000","last_change":"2019-12-11T16:33:08.637025+0000","last_active":"2019-12-11T16:33:08.659022+0000","last_peered":"2019-12-11T16:33:08.659022+0000","last_clean":"2019-12-11T16:33:08.659022+0000","last_became_active":"2019-12-11T16:33:08.616039+0000","last_became_peered":"2019-12-11T16:33:08.616039+0000","last_unstale":"2019-12-11T16:33:08.659022+0000","last_undegraded":"2019-12-11T16:33:08.659022+0000","last_fullsized":"2019-12-11T16:33:08.659022+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2019-12-11T16:33:09.034637+0000","last_change":"2019-12-11T16:33:06.972480+0000","last_active":"2019-12-11T16:33:09.034637+0000","last_peered":"2019-12-11T16:33:09.034637+0000","last_clean":"2019-12-11T16:33:09.034637+0000","last_became_active":"2019-12-11T16:33:06.933094+0000","last_became_peered":"2019-12-11T16:33:06.933094+0000","last_unstale":"2019-12-11T16:33:09.034637+0000","last_undegraded":"2019-12-11T16:33:09.034637+0000","last_fullsized":"2019-12-11T16:33:09.034637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-11T16:33:05.264613+0000","last_clean_scrub_stamp":"2019-12-11T16:33:05.264613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.397}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":12.989}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.150}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.012}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.928}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":33.831}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":36.432}]}]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.054}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.612}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.952}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":27.120}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.680}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":36.059}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.222}]}]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.131}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":17.005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.981}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":18.986}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":27.224}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":41.064}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.671}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.394}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.899}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.739}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.346}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.852}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.588}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.987}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":10.142}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":13.597}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":9.294}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":17.967}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.690}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.357}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":17.384}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.530}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.051}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.734}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.519}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.878}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.485}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":8.841}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":18.960}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.755}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.678}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.066}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":30.802}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.629}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048776,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323064,"statfs":{"total":96636764160,"available":95562817536,"internally_reserved":1073741824,"allocated":196608,"data_stored":64092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":11.126}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":12.295}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":10.322}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":18.862}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.815}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.620}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.933}]}]}]}} 2019-12-11T16:33:23.891 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-12-11T16:33:23.892 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-12-11T16:33:23.892 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2019-12-11T16:33:23.893 INFO:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2019-12-11T16:33:24.639 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:33:24.656 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2019-12-11T16:33:24.656 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2019-12-11T16:33:24.657 INFO:teuthology.run_tasks:Running task openssl_keys... 2019-12-11T16:33:24.667 INFO:teuthology.run_tasks:Running task rgw... 2019-12-11T16:33:24.725 DEBUG:tasks.rgw:config is {'client.0': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}} 2019-12-11T16:33:24.726 DEBUG:tasks.rgw:client list is ['client.0'] 2019-12-11T16:33:24.726 INFO:tasks.rgw:Creating data pools 2019-12-11T16:33:24.727 DEBUG:tasks.rgw:Obtaining remote for client client.0 2019-12-11T16:33:24.727 INFO:teuthology.orchestra.run.smithi079:> sudo ceph osd erasure-code-profile set client.0 crush-failure-domain=osd k=3 m=1 name=testprofile --cluster ceph 2019-12-11T16:33:26.803 INFO:teuthology.orchestra.run.smithi079:> sudo ceph osd pool create default.rgw.buckets.data 64 64 erasure client.0 --cluster ceph 2019-12-11T16:33:28.919 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'default.rgw.buckets.data' created 2019-12-11T16:33:28.933 INFO:teuthology.orchestra.run.smithi079:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2019-12-11T16:33:30.300 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2019-12-11T16:33:30.320 INFO:teuthology.orchestra.run.smithi079:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2019-12-11T16:33:31.323 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'default.rgw.buckets.index' created 2019-12-11T16:33:31.345 INFO:teuthology.orchestra.run.smithi079:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2019-12-11T16:33:32.364 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2019-12-11T16:33:32.381 DEBUG:tasks.rgw:Pools created 2019-12-11T16:33:32.381 INFO:tasks.rgw:Configuring compression type = random 2019-12-11T16:33:32.381 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2019-12-11T16:33:32.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2019-12-11T16:33:32.382 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2019-12-11T16:33:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:32.639+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:35.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:35.028+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:35.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:35.028+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:35.140 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:35.143+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:35.140 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:35.143+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:35.146 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:35.149+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:35.146 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:35.149+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:35.153 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:35.157+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:36.272 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.275+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:36.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.282+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:36.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.282+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:36.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.286+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:36.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.286+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:36.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.292+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:36.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.292+0000 7fedf4ce4ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:33:36.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.293+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:36.297 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.301+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:36.297 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.301+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:36.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.307+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:36.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.307+0000 7fedf4ce4ec0 10 Creating default zonegroup 2019-12-11T16:33:36.304 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.307+0000 7fedf4ce4ec0 20 get_system_obj_state: rctx=0x7ffd3569a420 obj=:.pools.avail state=0x55974dc9c270 s->prefetch_data=0 2019-12-11T16:33:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.310+0000 7fedf4ce4ec0 10 couldn't find old data placement pools config, setting up new ones for the zone 2019-12-11T16:33:36.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.484+0000 7fedf4ce4ec0 10 WARNING: store->list_zones() returned r=-2 2019-12-11T16:33:36.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.484+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:36.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:36.493+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:37.583 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:37.586+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:37.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:37.597+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:37.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:37.597+0000 7fedf4ce4ec0 10 could not read realm id: (2) No such file or directory 2019-12-11T16:33:37.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:37.597+0000 7fedf4ce4ec0 10 WARNING: failed to set zone as default, r=-22 2019-12-11T16:33:37.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:37.597+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:37.599 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:37.603+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:38.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.124+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:38.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.196+0000 7fedf4ce4ec0 20 rados_obj.operate() r=0 bl.length=805 2019-12-11T16:33:38.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.196+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:38.281 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.284+0000 7fedf4ce4ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-11T16:33:38.281 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.284+0000 7fedf4ce4ec0 20 zonegroup default 2019-12-11T16:33:38.281 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.284+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:38.291 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.294+0000 7fedf4ce4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:38.291 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.294+0000 7fedf4ce4ec0 10 Cannot find current period zone using local zone 2019-12-11T16:33:38.291 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.294+0000 7fedf4ce4ec0 10 Using default name default 2019-12-11T16:33:38.292 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.294+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:38.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.342+0000 7fedf4ce4ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:38.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.342+0000 7fedf4ce4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:38.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.360+0000 7fedf4ce4ec0 20 rados_obj.operate() r=0 bl.length=805 2019-12-11T16:33:38.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.360+0000 7fedf4ce4ec0 20 zone default found 2019-12-11T16:33:38.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.360+0000 7fedf4ce4ec0 20 started sync module instance, tier type = 2019-12-11T16:33:38.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:38.360+0000 7fedf4ce4ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:33:39.800 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:39.804+0000 7fede270f700 0 -- 172.21.15.79:0/485102240 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x55974dc8a290 msgr2=0x55974dc8c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:33:42.516 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:42.520+0000 7fede1f0e700 0 -- 172.21.15.79:0/485102240 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x55974dc992b0 msgr2=0x55974dc9b6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:33:43.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.182+0000 7fedf4ce4ec0 20 add_watcher() i=0 2019-12-11T16:33:43.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.182+0000 7fedf4ce4ec0 20 add_watcher() i=1 2019-12-11T16:33:43.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.182+0000 7fedf4ce4ec0 20 add_watcher() i=2 2019-12-11T16:33:43.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.182+0000 7fedf4ce4ec0 20 add_watcher() i=3 2019-12-11T16:33:43.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.182+0000 7fedf4ce4ec0 20 add_watcher() i=4 2019-12-11T16:33:43.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.182+0000 7fedf4ce4ec0 20 add_watcher() i=5 2019-12-11T16:33:43.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.182+0000 7fedf4ce4ec0 20 add_watcher() i=6 2019-12-11T16:33:43.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.216+0000 7fedf4ce4ec0 20 add_watcher() i=7 2019-12-11T16:33:43.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.216+0000 7fedf4ce4ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:33:43.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:43.216+0000 7fedb27dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:33:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:46.955+0000 7fedf4ce4ec0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2019-12-11T16:33:51.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:51.847+0000 7fedf4ce4ec0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.log 2019-12-11T16:33:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:55.606+0000 7fed20ff9700 20 reqs_thread_entry: start 2019-12-11T16:33:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:55.607+0000 7fed127fc700 20 reqs_thread_entry: start 2019-12-11T16:33:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:55.607+0000 7fedf4ce4ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:33:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:55.608+0000 7fed10ff9700 20 reqs_thread_entry: start 2019-12-11T16:33:56.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.012+0000 7fedf4ce4ec0 20 remove_watcher() i=0 2019-12-11T16:33:56.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.012+0000 7fedf4ce4ec0 2 removed watcher, disabling cache 2019-12-11T16:33:56.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.078+0000 7fedf4ce4ec0 20 remove_watcher() i=1 2019-12-11T16:33:56.113 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.117+0000 7fedf4ce4ec0 20 remove_watcher() i=2 2019-12-11T16:33:56.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.190+0000 7fedf4ce4ec0 20 remove_watcher() i=3 2019-12-11T16:33:56.281 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.285+0000 7fedf4ce4ec0 20 remove_watcher() i=4 2019-12-11T16:33:56.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.349+0000 7fedf4ce4ec0 20 remove_watcher() i=5 2019-12-11T16:33:56.440 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.443+0000 7fedf4ce4ec0 20 remove_watcher() i=6 2019-12-11T16:33:56.521 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.525+0000 7fedf4ce4ec0 20 remove_watcher() i=7 2019-12-11T16:33:56.531 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2019-12-11T16:33:56.531 DEBUG:tasks.util.rgw: json result: [] 2019-12-11T16:33:56.532 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2019-12-11T16:33:56.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2019-12-11T16:33:56.532 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random 2019-12-11T16:33:56.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.779+0000 7ff05dffb700 20 reqs_thread_entry: start 2019-12-11T16:33:56.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.779+0000 7ff128600ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:56.792 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.793+0000 7ff128600ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:56.792 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.793+0000 7ff128600ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:56.814 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.818+0000 7ff128600ec0 20 rados_obj.operate() r=0 bl.length=805 2019-12-11T16:33:56.814 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.818+0000 7ff128600ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:56.821 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.824+0000 7ff128600ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:56.821 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.824+0000 7ff128600ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:56.832 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.836+0000 7ff128600ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:56.833 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.836+0000 7ff128600ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:56.857 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:56.862+0000 7ff128600ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-11T16:33:57.024 INFO:teuthology.orchestra.run.smithi079.stdout:{"id":"14af9c21-7f56-4de3-9a52-895b5fee19a0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2019-12-11T16:33:57.025 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'roles_pool': u'default.rgw.meta:roles', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'user_email_pool': u'default.rgw.meta:users.email', u'realm_id': u'', u'placement_pools': [{u'val': {u'storage_classes': {u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'id': u'14af9c21-7f56-4de3-9a52-895b5fee19a0', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2019-12-11T16:33:57.025 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2019-12-11T16:33:57.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2019-12-11T16:33:57.026 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2019-12-11T16:33:57.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.218+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.242 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.245+0000 7f2e42eb1ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:57.242 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.245+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.255+0000 7f2e42eb1ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:57.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.255+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.260+0000 7f2e42eb1ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:57.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.261+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.269+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:57.526 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.529+0000 7f2e42eb1ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:33:57.526 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.529+0000 7f2e42eb1ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:33:57.526 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.529+0000 7f2e42eb1ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:33:57.527 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.529+0000 7f2e42eb1ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:33:57.527 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.529+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.544 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.548+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:57.544 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.548+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.569 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.572+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-11T16:33:57.569 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.572+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.603+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:57.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.603+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.613 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.617+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-11T16:33:57.614 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.617+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.622 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.625+0000 7f2e42eb1ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:57.622 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.625+0000 7f2e42eb1ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:33:57.622 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.625+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.628 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.632+0000 7f2e42eb1ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:57.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.632+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.651+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:57.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.651+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.660 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.664+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-11T16:33:57.660 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.664+0000 7f2e42eb1ec0 20 zonegroup default 2019-12-11T16:33:57.660 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.664+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.669+0000 7f2e42eb1ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:57.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.669+0000 7f2e42eb1ec0 10 Cannot find current period zone using local zone 2019-12-11T16:33:57.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.669+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.674 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.677+0000 7f2e42eb1ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:33:57.674 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.677+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.692+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:33:57.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.692+0000 7f2e42eb1ec0 20 rados->read ofs=0 len=0 2019-12-11T16:33:57.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.709+0000 7f2e42eb1ec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-11T16:33:57.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.709+0000 7f2e42eb1ec0 20 zone default found 2019-12-11T16:33:57.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.709+0000 7f2e42eb1ec0 20 started sync module instance, tier type = 2019-12-11T16:33:57.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:57.709+0000 7f2e42eb1ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:33:58.142 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.144+0000 7f2e300db700 0 -- 172.21.15.79:0/2433275239 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x564585d74740 msgr2=0x564585d76b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:33:58.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.208+0000 7f2e42eb1ec0 20 add_watcher() i=0 2019-12-11T16:33:58.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.208+0000 7f2e42eb1ec0 20 add_watcher() i=1 2019-12-11T16:33:58.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.208+0000 7f2e42eb1ec0 20 add_watcher() i=2 2019-12-11T16:33:58.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.208+0000 7f2e42eb1ec0 20 add_watcher() i=3 2019-12-11T16:33:58.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.208+0000 7f2e42eb1ec0 20 add_watcher() i=4 2019-12-11T16:33:58.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.208+0000 7f2e42eb1ec0 20 add_watcher() i=5 2019-12-11T16:33:58.225 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.229+0000 7f2e42eb1ec0 20 add_watcher() i=6 2019-12-11T16:33:58.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.250+0000 7f2e42eb1ec0 20 add_watcher() i=7 2019-12-11T16:33:58.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.250+0000 7f2e42eb1ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:33:58.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.250+0000 7f2e00fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:33:58.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:33:58.281+0000 7f2e308dc700 0 -- 172.21.15.79:0/2489667528 >> v2:172.21.15.79:3300/0 conn(0x564585d86df0 msgr2=0x564585d87200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-11T16:34:00.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:00.742+0000 7f2d72ffd700 20 reqs_thread_entry: start 2019-12-11T16:34:00.749 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:00.743+0000 7f2d70ff9700 20 reqs_thread_entry: start 2019-12-11T16:34:00.750 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:00.744+0000 7f2e42eb1ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:34:00.751 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:00.745+0000 7f2d66ffd700 20 reqs_thread_entry: start 2019-12-11T16:34:00.978 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:00.982+0000 7f2e42eb1ec0 20 remove_watcher() i=0 2019-12-11T16:34:00.978 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:00.982+0000 7f2e42eb1ec0 2 removed watcher, disabling cache 2019-12-11T16:34:01.056 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.060+0000 7f2e42eb1ec0 20 remove_watcher() i=1 2019-12-11T16:34:01.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.114+0000 7f2e42eb1ec0 20 remove_watcher() i=2 2019-12-11T16:34:01.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.164+0000 7f2e42eb1ec0 20 remove_watcher() i=3 2019-12-11T16:34:01.227 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.230+0000 7f2e42eb1ec0 20 remove_watcher() i=4 2019-12-11T16:34:01.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.279+0000 7f2e42eb1ec0 20 remove_watcher() i=5 2019-12-11T16:34:01.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.333+0000 7f2e42eb1ec0 20 remove_watcher() i=6 2019-12-11T16:34:01.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.392+0000 7f2e42eb1ec0 20 remove_watcher() i=7 2019-12-11T16:34:01.404 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2019-12-11T16:34:01.404 DEBUG:tasks.util.rgw: json result: [] 2019-12-11T16:34:01.405 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2019-12-11T16:34:01.405 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2019-12-11T16:34:01.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2019-12-11T16:34:01.405 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2019-12-11T16:34:01.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.687+0000 7fe788ff9700 20 reqs_thread_entry: start 2019-12-11T16:34:01.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.687+0000 7fe84b867ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:01.708 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.712+0000 7fe84b867ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:34:01.708 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.712+0000 7fe84b867ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:01.716 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.720+0000 7fe84b867ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:34:01.716 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.720+0000 7fe84b867ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:01.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.745+0000 7fe84b867ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-11T16:34:01.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.745+0000 7fe84b867ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:01.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:01.768+0000 7fe84b867ec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-11T16:34:01.861 INFO:teuthology.orchestra.run.smithi079.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["LUKEWARM","STANDARD"]}}] 2019-12-11T16:34:01.862 DEBUG:tasks.util.rgw: json result: [{u'val': {u'storage_classes': [u'LUKEWARM', u'STANDARD'], u'name': u'default-placement', u'tags': []}, u'key': u'default-placement'}] 2019-12-11T16:34:01.862 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2019-12-11T16:34:01.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2019-12-11T16:34:01.862 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2019-12-11T16:34:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.084+0000 7fdbc330eec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.084+0000 7fdaf7fff700 20 reqs_thread_entry: start 2019-12-11T16:34:02.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.096+0000 7fdbc330eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:34:02.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.096+0000 7fdbc330eec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.123 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.122+0000 7fdbc330eec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-11T16:34:02.123 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.122+0000 7fdbc330eec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.128+0000 7fdbc330eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:34:02.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.128+0000 7fdbc330eec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.135 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.139+0000 7fdbc330eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:34:02.135 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.139+0000 7fdbc330eec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.165+0000 7fdbc330eec0 20 rados_obj.operate() r=0 bl.length=372 2019-12-11T16:34:02.247 INFO:teuthology.orchestra.run.smithi079.stdout:{"id":"14af9c21-7f56-4de3-9a52-895b5fee19a0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2019-12-11T16:34:02.247 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'roles_pool': u'default.rgw.meta:roles', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'user_email_pool': u'default.rgw.meta:users.email', u'realm_id': u'', u'placement_pools': [{u'val': {u'storage_classes': {u'LUKEWARM': {u'data_pool': u'default.rgw.buckets.data.lukewarm'}, u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'id': u'14af9c21-7f56-4de3-9a52-895b5fee19a0', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2019-12-11T16:34:02.247 INFO:tasks.rgw:Configuring storage class type = FROZEN 2019-12-11T16:34:02.247 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2019-12-11T16:34:02.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2019-12-11T16:34:02.248 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2019-12-11T16:34:02.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.469+0000 7f20ab7fe700 20 reqs_thread_entry: start 2019-12-11T16:34:02.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.469+0000 7f217a8ccec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.477 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.481+0000 7f217a8ccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:34:02.478 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.481+0000 7f217a8ccec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.493+0000 7f217a8ccec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:34:02.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.493+0000 7f217a8ccec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.512+0000 7f217a8ccec0 20 rados_obj.operate() r=0 bl.length=372 2019-12-11T16:34:02.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.512+0000 7f217a8ccec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.550+0000 7f217a8ccec0 20 rados_obj.operate() r=0 bl.length=888 2019-12-11T16:34:02.601 INFO:teuthology.orchestra.run.smithi079.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2019-12-11T16:34:02.602 DEBUG:tasks.util.rgw: json result: [{u'val': {u'storage_classes': [u'FROZEN', u'LUKEWARM', u'STANDARD'], u'name': u'default-placement', u'tags': []}, u'key': u'default-placement'}] 2019-12-11T16:34:02.602 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2019-12-11T16:34:02.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2019-12-11T16:34:02.602 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2019-12-11T16:34:02.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.760+0000 7f2b627fc700 20 reqs_thread_entry: start 2019-12-11T16:34:02.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.760+0000 7f2c2d224ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.785+0000 7f2c2d224ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:34:02.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.785+0000 7f2c2d224ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.798+0000 7f2c2d224ec0 20 rados_obj.operate() r=0 bl.length=888 2019-12-11T16:34:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.798+0000 7f2c2d224ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.804+0000 7f2c2d224ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:34:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.804+0000 7f2c2d224ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.813 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.813+0000 7f2c2d224ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:34:02.813 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.813+0000 7f2c2d224ec0 20 rados->read ofs=0 len=0 2019-12-11T16:34:02.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:34:02.826+0000 7f2c2d224ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:34:02.869 INFO:teuthology.orchestra.run.smithi079.stdout:{"id":"14af9c21-7f56-4de3-9a52-895b5fee19a0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2019-12-11T16:34:03.124 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'roles_pool': u'default.rgw.meta:roles', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'user_email_pool': u'default.rgw.meta:users.email', u'realm_id': u'', u'placement_pools': [{u'val': {u'storage_classes': {u'FROZEN': {u'data_pool': u'default.rgw.buckets.data.frozen'}, u'LUKEWARM': {u'data_pool': u'default.rgw.buckets.data.lukewarm'}, u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'id': u'14af9c21-7f56-4de3-9a52-895b5fee19a0', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2019-12-11T16:34:03.125 INFO:tasks.rgw:Starting rgw... 2019-12-11T16:34:03.125 INFO:tasks.rgw:rgw client.0 config is {'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2019-12-11T16:34:03.125 INFO:tasks.rgw:Using civetweb as radosgw frontend 2019-12-11T16:34:03.126 DEBUG:teuthology.misc:running ceph.client.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/ceph.client.0.log', '--time-stamp=yes', '--tool=memcheck', '--max-threads=1024'] 2019-12-11T16:34:03.126 INFO:tasks.rgw.client.0:Restarting daemon 2019-12-11T16:34:03.127 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.0.log --time-stamp=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'civetweb port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2019-12-11T16:34:03.134 INFO:tasks.rgw.client.0:Started 2019-12-11T16:34:03.134 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi079.front.sepia.ceph.com:80/ 2019-12-11T16:34:03.135 INFO:teuthology.orchestra.run.smithi079:> curl http://smithi079.front.sepia.ceph.com:80/ 2019-12-11T16:34:03.232 INFO:teuthology.orchestra.run.smithi079.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-11T16:34:03.232 INFO:teuthology.orchestra.run.smithi079.stderr: Dload Upload Total Spent Left Speed 2019-12-11T16:34:03.236 INFO:teuthology.orchestra.run.smithi079.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi079.front.sepia.ceph.com:80; Connection refused 2019-12-11T16:34:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-11T16:34:04.237 INFO:teuthology.orchestra.run.smithi079:> curl http://smithi079.front.sepia.ceph.com:80/ 2019-12-11T16:34:04.301 INFO:teuthology.orchestra.run.smithi079.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-11T16:34:04.303 INFO:teuthology.orchestra.run.smithi079.stderr: Dload Upload Total Spent Left Speed 2019-12-11T16:34:04.305 INFO:teuthology.orchestra.run.smithi079.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi079.front.sepia.ceph.com:80; Connection refused 2019-12-11T16:34:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-11T16:34:06.306 INFO:teuthology.orchestra.run.smithi079:> curl http://smithi079.front.sepia.ceph.com:80/ 2019-12-11T16:34:06.371 INFO:teuthology.orchestra.run.smithi079.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-11T16:34:06.372 INFO:teuthology.orchestra.run.smithi079.stderr: Dload Upload Total Spent Left Speed 2019-12-11T16:34:06.375 INFO:teuthology.orchestra.run.smithi079.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi079.front.sepia.ceph.com:80; Connection refused 2019-12-11T16:34:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-11T16:34:10.377 INFO:teuthology.orchestra.run.smithi079:> curl http://smithi079.front.sepia.ceph.com:80/ 2019-12-11T16:34:10.467 INFO:teuthology.orchestra.run.smithi079.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-11T16:34:10.467 INFO:teuthology.orchestra.run.smithi079.stderr: Dload Upload Total Spent Left Speed 2019-12-11T16:34:10.470 INFO:teuthology.orchestra.run.smithi079.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi079.front.sepia.ceph.com:80; Connection refused 2019-12-11T16:34:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-11T16:34:18.473 INFO:teuthology.orchestra.run.smithi079:> curl http://smithi079.front.sepia.ceph.com:80/ 2019-12-11T16:34:18.535 INFO:teuthology.orchestra.run.smithi079.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-11T16:34:18.535 INFO:teuthology.orchestra.run.smithi079.stderr: Dload Upload Total Spent Left Speed 2019-12-11T16:34:18.884 INFO:teuthology.orchestra.run.smithi079.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 609 0 --:--:-- --:--:-- --:--:-- 611 2019-12-11T16:34:18.884 INFO:teuthology.orchestra.run.smithi079.stdout:anonymous 2019-12-11T16:34:18.885 INFO:teuthology.run_tasks:Running task workunit... 2019-12-11T16:34:18.904 INFO:tasks.workunit:Pulling workunits from ref 1b78fd04fb1980223b57f0a3f644592e4e26b518 2019-12-11T16:34:18.904 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-12-11T16:34:18.905 INFO:teuthology.orchestra.run.smithi079:> stat -- /home/ubuntu/cephtest/mnt.0 2019-12-11T16:34:18.946 INFO:teuthology.orchestra.run.smithi079.stderr:stat: cannot stat ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2019-12-11T16:34:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-11T16:34:18.947 INFO:teuthology.orchestra.run.smithi079:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-12-11T16:34:19.045 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2019-12-11T16:34:19.045 INFO:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2019-12-11T16:34:19.091 INFO:tasks.workunit:timeout=3h 2019-12-11T16:34:19.091 INFO:tasks.workunit:cleanup=True 2019-12-11T16:34:19.091 INFO:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1b78fd04fb1980223b57f0a3f644592e4e26b518 2019-12-11T16:34:19.403 INFO:tasks.workunit.client.0.smithi079.stdout:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-12-11T16:35:24.549 INFO:tasks.workunit.client.0.smithi079.stderr:Note: checking out '1b78fd04fb1980223b57f0a3f644592e4e26b518'. 2019-12-11T16:35:24.549 INFO:tasks.workunit.client.0.smithi079.stderr: 2019-12-11T16:35:24.549 INFO:tasks.workunit.client.0.smithi079.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-12-11T16:35:24.550 INFO:tasks.workunit.client.0.smithi079.stderr:changes and commit them, and you can discard any commits you make in this 2019-12-11T16:35:24.550 INFO:tasks.workunit.client.0.smithi079.stderr:state without impacting any branches by performing another checkout. 2019-12-11T16:35:24.550 INFO:tasks.workunit.client.0.smithi079.stderr: 2019-12-11T16:35:24.550 INFO:tasks.workunit.client.0.smithi079.stderr:If you want to create a new branch to retain commits you create, you may 2019-12-11T16:35:24.550 INFO:tasks.workunit.client.0.smithi079.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-12-11T16:35:24.550 INFO:tasks.workunit.client.0.smithi079.stderr: 2019-12-11T16:35:24.550 INFO:tasks.workunit.client.0.smithi079.stderr: git checkout -b new_branch_name 2019-12-11T16:35:24.551 INFO:tasks.workunit.client.0.smithi079.stderr: 2019-12-11T16:35:24.551 INFO:tasks.workunit.client.0.smithi079.stderr:HEAD is now at 1b78fd0... Merge branch 'tmp/public-access-qa' of https://github.com/theanalyst/ceph into wip-abhi-testing-2019-12-10-1759 2019-12-11T16:35:24.552 INFO:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-12-11T16:35:24.607 INFO:tasks.workunit.client.0.smithi079.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2019-12-11T16:35:24.609 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-11T16:35:24.610 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2019-12-11T16:35:25.057 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2019-12-11T16:35:25.157 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2019-12-11T16:35:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-11T16:35:25.211 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-11T16:35:25.211 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2019-12-11T16:35:25.250 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-11T16:35:25.299 DEBUG:teuthology.orchestra.remote:smithi079:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2019-12-11T16:35:25.347 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2019-12-11T16:35:25.347 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2019-12-11T16:35:25.347 INFO:teuthology.orchestra.run.smithi079:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1b78fd04fb1980223b57f0a3f644592e4e26b518 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2019-12-11T16:35:25.401 INFO:tasks.workunit.client.0.smithi079.stdout:Running main() from gmock_main.cc 2019-12-11T16:35:25.401 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] Running 11 tests from 1 test suite. 2019-12-11T16:35:25.401 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment set-up. 2019-12-11T16:35:25.401 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 11 tests from ClsLock 2019-12-11T16:35:25.403 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestMultiLocking 2019-12-11T16:35:27.527 INFO:tasks.workunit.client.0.smithi079.stdout:lock: mylock 2019-12-11T16:35:27.527 INFO:tasks.workunit.client.0.smithi079.stdout: lock_type: exclusive 2019-12-11T16:35:27.527 INFO:tasks.workunit.client.0.smithi079.stdout: tag: 2019-12-11T16:35:27.528 INFO:tasks.workunit.client.0.smithi079.stdout: lockers: 2019-12-11T16:35:27.528 INFO:tasks.workunit.client.0.smithi079.stdout: client.4343 expiration=2019-12-11T16:37:27.359211+0000 addr=v1:172.21.15.79:0/693841186 cookie= 2019-12-11T16:35:27.692 INFO:tasks.workunit.client.0.smithi079.stdout:lock: mylock 2019-12-11T16:35:27.692 INFO:tasks.workunit.client.0.smithi079.stdout: lock_type: shared 2019-12-11T16:35:27.692 INFO:tasks.workunit.client.0.smithi079.stdout: tag: 2019-12-11T16:35:27.692 INFO:tasks.workunit.client.0.smithi079.stdout: lockers: 2019-12-11T16:35:27.693 INFO:tasks.workunit.client.0.smithi079.stdout: client.4343 expiration=2019-12-11T16:37:27.642485+0000 addr=v1:172.21.15.79:0/693841186 cookie= 2019-12-11T16:35:27.693 INFO:tasks.workunit.client.0.smithi079.stdout: client.4349 expiration=0.000000 addr=v1:172.21.15.79:0/725603224 cookie= 2019-12-11T16:35:27.754 INFO:tasks.workunit.client.0.smithi079.stdout:lock: mylock 2019-12-11T16:35:27.754 INFO:tasks.workunit.client.0.smithi079.stdout: lock_type: shared 2019-12-11T16:35:27.754 INFO:tasks.workunit.client.0.smithi079.stdout: tag: 2019-12-11T16:35:27.754 INFO:tasks.workunit.client.0.smithi079.stdout: lockers: 2019-12-11T16:35:27.754 INFO:tasks.workunit.client.0.smithi079.stdout: client.4349 expiration=0.000000 addr=v1:172.21.15.79:0/725603224 cookie= 2019-12-11T16:35:29.122 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestMultiLocking (3719 ms) 2019-12-11T16:35:29.122 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestMeta 2019-12-11T16:35:32.629 INFO:tasks.workunit.client.0.smithi079.stdout:lock: mylock 2019-12-11T16:35:32.629 INFO:tasks.workunit.client.0.smithi079.stdout: lock_type: shared 2019-12-11T16:35:32.629 INFO:tasks.workunit.client.0.smithi079.stdout: tag: 2019-12-11T16:35:32.629 INFO:tasks.workunit.client.0.smithi079.stdout: lockers: 2019-12-11T16:35:32.629 INFO:tasks.workunit.client.0.smithi079.stdout: client.4522 expiration=0.000000 addr=v1:172.21.15.79:0/1232246564 cookie= 2019-12-11T16:35:32.727 INFO:tasks.workunit.client.0.smithi079.stdout:lock: mylock 2019-12-11T16:35:32.727 INFO:tasks.workunit.client.0.smithi079.stdout: lock_type: exclusive 2019-12-11T16:35:32.727 INFO:tasks.workunit.client.0.smithi079.stdout: tag: new_tag 2019-12-11T16:35:32.727 INFO:tasks.workunit.client.0.smithi079.stdout: lockers: 2019-12-11T16:35:32.727 INFO:tasks.workunit.client.0.smithi079.stdout: client.4513 expiration=0.000000 addr=v1:172.21.15.79:0/921878543 cookie= 2019-12-11T16:35:33.720 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestMeta (4601 ms) 2019-12-11T16:35:33.720 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestCookie 2019-12-11T16:35:36.560 INFO:tasks.workunit.client.0.smithi079.stdout:lock: mylock 2019-12-11T16:35:36.561 INFO:tasks.workunit.client.0.smithi079.stdout: lock_type: exclusive 2019-12-11T16:35:36.561 INFO:tasks.workunit.client.0.smithi079.stdout: tag: 2019-12-11T16:35:36.561 INFO:tasks.workunit.client.0.smithi079.stdout: lockers: 2019-12-11T16:35:36.659 INFO:tasks.workunit.client.0.smithi079.stdout:lock: mylock 2019-12-11T16:35:36.660 INFO:tasks.workunit.client.0.smithi079.stdout: lock_type: shared 2019-12-11T16:35:36.660 INFO:tasks.workunit.client.0.smithi079.stdout: tag: 2019-12-11T16:35:36.660 INFO:tasks.workunit.client.0.smithi079.stdout: lockers: 2019-12-11T16:35:36.660 INFO:tasks.workunit.client.0.smithi079.stdout: client.4563 expiration=0.000000 addr=v1:172.21.15.79:0/3669256208 cookie= 2019-12-11T16:35:36.660 INFO:tasks.workunit.client.0.smithi079.stdout: client.4563 expiration=0.000000 addr=v1:172.21.15.79:0/3669256208 cookie=new cookie 2019-12-11T16:35:37.101 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestCookie (3377 ms) 2019-12-11T16:35:37.101 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestMultipleLocks 2019-12-11T16:35:40.649 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestMultipleLocks (3551 ms) 2019-12-11T16:35:40.649 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestLockDuration 2019-12-11T16:35:48.820 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestLockDuration (8170 ms) 2019-12-11T16:35:48.820 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestAssertLocked 2019-12-11T16:35:52.353 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestAssertLocked (3529 ms) 2019-12-11T16:35:52.354 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestSetCookie 2019-12-11T16:35:56.714 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestSetCookie (4366 ms) 2019-12-11T16:35:56.714 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestRenew 2019-12-11T16:36:18.865 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestRenew (22151 ms) 2019-12-11T16:36:18.865 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2019-12-11T16:36:26.865 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7999 ms) 2019-12-11T16:36:26.865 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2019-12-11T16:36:35.437 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (8573 ms) 2019-12-11T16:36:35.437 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2019-12-11T16:36:44.432 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8996 ms) 2019-12-11T16:36:44.432 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 11 tests from ClsLock (79033 ms total) 2019-12-11T16:36:44.432 INFO:tasks.workunit.client.0.smithi079.stdout: 2019-12-11T16:36:44.433 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment tear-down 2019-12-11T16:36:44.433 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] 11 tests from 1 test suite ran. (79033 ms total) 2019-12-11T16:36:44.433 INFO:tasks.workunit.client.0.smithi079.stdout:[ PASSED ] 11 tests. 2019-12-11T16:36:44.434 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-11T16:36:44.435 INFO:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-11T16:36:44.601 INFO:tasks.workunit:Running workunits matching cls/test_cls_log.sh on client.0... 2019-12-11T16:36:44.610 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2019-12-11T16:36:44.611 INFO:teuthology.orchestra.run.smithi079:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1b78fd04fb1980223b57f0a3f644592e4e26b518 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2019-12-11T16:36:44.693 INFO:tasks.workunit.client.0.smithi079.stdout:Running main() from gmock_main.cc 2019-12-11T16:36:44.693 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] Running 4 tests from 1 test suite. 2019-12-11T16:36:44.693 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment set-up. 2019-12-11T16:36:44.694 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 4 tests from cls_log 2019-12-11T16:36:44.694 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_log.test_log_add_same_time 2019-12-11T16:36:47.859 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_log.test_log_add_same_time (3170 ms) 2019-12-11T16:36:47.859 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_log.test_log_add_different_time 2019-12-11T16:36:51.407 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_log.test_log_add_different_time (3548 ms) 2019-12-11T16:36:51.408 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_log.trim_by_time 2019-12-11T16:36:56.878 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_log.trim_by_time (5471 ms) 2019-12-11T16:36:56.878 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_log.trim_by_marker 2019-12-11T16:37:01.164 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_log.trim_by_marker (4284 ms) 2019-12-11T16:37:01.164 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 4 tests from cls_log (16473 ms total) 2019-12-11T16:37:01.164 INFO:tasks.workunit.client.0.smithi079.stdout: 2019-12-11T16:37:01.164 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment tear-down 2019-12-11T16:37:01.165 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] 4 tests from 1 test suite ran. (16474 ms total) 2019-12-11T16:37:01.165 INFO:tasks.workunit.client.0.smithi079.stdout:[ PASSED ] 4 tests. 2019-12-11T16:37:01.165 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-11T16:37:01.165 INFO:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-11T16:37:01.234 INFO:tasks.workunit:Running workunits matching cls/test_cls_refcount.sh on client.0... 2019-12-11T16:37:01.234 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2019-12-11T16:37:01.235 INFO:teuthology.orchestra.run.smithi079:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1b78fd04fb1980223b57f0a3f644592e4e26b518 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2019-12-11T16:37:01.298 INFO:tasks.workunit.client.0.smithi079.stdout:Running main() from gmock_main.cc 2019-12-11T16:37:01.298 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] Running 5 tests from 1 test suite. 2019-12-11T16:37:01.299 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment set-up. 2019-12-11T16:37:01.299 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 5 tests from cls_rgw 2019-12-11T16:37:01.299 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.test_implicit 2019-12-11T16:37:04.856 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.test_implicit (3558 ms) 2019-12-11T16:37:04.856 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.test_implicit_idempotent 2019-12-11T16:37:08.160 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.test_implicit_idempotent (3303 ms) 2019-12-11T16:37:08.161 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.test_put_snap 2019-12-11T16:37:17.123 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T16:37:17.080+0000 38794700 -1 osd.2 pg_epoch: 106 pg[25.6( v 105'4 (0'0,105'4] local-lis/les=102/103 n=2 ec=102/102 lis/c=102/102 les/c/f=103/103/0 sis=102) [2,1] r=0 lpr=102 crt=104'3 lcod 104'3 mlcod 104'3 active+clean+snaptrim trimq=[1~1]] removing snap head 2019-12-11T16:37:18.569 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.test_put_snap (10406 ms) 2019-12-11T16:37:18.569 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.test_explicit 2019-12-11T16:37:22.168 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.test_explicit (3604 ms) 2019-12-11T16:37:22.169 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.set 2019-12-11T16:37:25.668 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.set (3499 ms) 2019-12-11T16:37:25.668 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 5 tests from cls_rgw (24370 ms total) 2019-12-11T16:37:25.668 INFO:tasks.workunit.client.0.smithi079.stdout: 2019-12-11T16:37:25.668 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment tear-down 2019-12-11T16:37:25.668 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] 5 tests from 1 test suite ran. (24370 ms total) 2019-12-11T16:37:25.669 INFO:tasks.workunit.client.0.smithi079.stdout:[ PASSED ] 5 tests. 2019-12-11T16:37:25.671 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-11T16:37:25.671 INFO:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-11T16:37:25.790 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw.sh on client.0... 2019-12-11T16:37:25.790 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2019-12-11T16:37:25.791 INFO:teuthology.orchestra.run.smithi079:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1b78fd04fb1980223b57f0a3f644592e4e26b518 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2019-12-11T16:37:25.851 INFO:tasks.workunit.client.0.smithi079.stdout:Running main() from gmock_main.cc 2019-12-11T16:37:25.851 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] Running 13 tests from 1 test suite. 2019-12-11T16:37:25.851 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment set-up. 2019-12-11T16:37:25.851 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 13 tests from cls_rgw 2019-12-11T16:37:27.760 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.index_basic 2019-12-11T16:37:29.346 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.index_basic (1586 ms) 2019-12-11T16:37:29.346 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2019-12-11T16:37:30.954 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (1609 ms) 2019-12-11T16:37:30.954 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.index_remove_object 2019-12-11T16:37:32.730 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.index_remove_object (1774 ms) 2019-12-11T16:37:32.731 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.index_suggest 2019-12-11T16:37:58.097 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.index_suggest (25369 ms) 2019-12-11T16:37:58.097 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.index_list 2019-12-11T16:38:00.047 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.index_list (1950 ms) 2019-12-11T16:38:00.047 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.bi_list 2019-12-11T16:38:03.220 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.bi_list (3173 ms) 2019-12-11T16:38:03.220 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.gc_set 2019-12-11T16:38:03.649 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.gc_set (429 ms) 2019-12-11T16:38:03.649 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.gc_list 2019-12-11T16:38:04.124 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.gc_list (475 ms) 2019-12-11T16:38:04.125 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.gc_defer 2019-12-11T16:38:12.909 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.gc_defer (8784 ms) 2019-12-11T16:38:12.909 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.usage_basic 2019-12-11T16:38:20.467 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.usage_basic (7557 ms) 2019-12-11T16:38:20.467 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2019-12-11T16:38:20.535 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.usage_clear_no_obj (70 ms) 2019-12-11T16:38:20.535 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.usage_clear 2019-12-11T16:38:30.305 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.usage_clear (9770 ms) 2019-12-11T16:38:30.305 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] cls_rgw.bi_log_trim 2019-12-11T16:38:32.190 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] cls_rgw.bi_log_trim (1885 ms) 2019-12-11T16:38:32.919 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 13 tests from cls_rgw (64431 ms total) 2019-12-11T16:38:32.920 INFO:tasks.workunit.client.0.smithi079.stdout: 2019-12-11T16:38:32.920 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment tear-down 2019-12-11T16:38:32.920 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] 13 tests from 1 test suite ran. (67067 ms total) 2019-12-11T16:38:32.920 INFO:tasks.workunit.client.0.smithi079.stdout:[ PASSED ] 13 tests. 2019-12-11T16:38:32.921 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-11T16:38:32.921 INFO:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-11T16:38:33.040 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_gc_log.sh on client.0... 2019-12-11T16:38:33.040 INFO:tasks.workunit:Running workunit rgw/test_rgw_gc_log.sh... 2019-12-11T16:38:33.040 INFO:teuthology.orchestra.run.smithi079:workunit test rgw/test_rgw_gc_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1b78fd04fb1980223b57f0a3f644592e4e26b518 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_gc_log.sh 2019-12-11T16:38:33.135 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] Running 6 tests from 1 test suite. 2019-12-11T16:38:33.135 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment set-up. 2019-12-11T16:38:35.292 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 6 tests from rgw_gc_log 2019-12-11T16:38:35.292 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] rgw_gc_log.init_existing_queue 2019-12-11T16:38:35.682 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] rgw_gc_log.init_existing_queue (391 ms) 2019-12-11T16:38:35.683 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] rgw_gc_log.init_existing_omap 2019-12-11T16:38:35.899 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] rgw_gc_log.init_existing_omap (216 ms) 2019-12-11T16:38:35.899 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] rgw_gc_log.enqueue1_after_init 2019-12-11T16:38:36.030 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] rgw_gc_log.enqueue1_after_init (131 ms) 2019-12-11T16:38:36.030 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] rgw_gc_log.enqueue2_before_init 2019-12-11T16:38:36.121 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] rgw_gc_log.enqueue2_before_init (91 ms) 2019-12-11T16:38:36.121 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] rgw_gc_log.defer1_after_init 2019-12-11T16:38:36.242 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] rgw_gc_log.defer1_after_init (122 ms) 2019-12-11T16:38:36.242 INFO:tasks.workunit.client.0.smithi079.stdout:[ RUN ] rgw_gc_log.defer2_before_init 2019-12-11T16:38:36.393 INFO:tasks.workunit.client.0.smithi079.stdout:[ OK ] rgw_gc_log.defer2_before_init (151 ms) 2019-12-11T16:38:36.394 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] 6 tests from rgw_gc_log (1102 ms total) 2019-12-11T16:38:36.394 INFO:tasks.workunit.client.0.smithi079.stdout: 2019-12-11T16:38:36.394 INFO:tasks.workunit.client.0.smithi079.stdout:[----------] Global test environment tear-down 2019-12-11T16:38:37.583 INFO:tasks.workunit.client.0.smithi079.stdout:[==========] 6 tests from 1 test suite ran. (4449 ms total) 2019-12-11T16:38:37.583 INFO:tasks.workunit.client.0.smithi079.stdout:[ PASSED ] 6 tests. 2019-12-11T16:38:37.586 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-11T16:38:37.587 INFO:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-11T16:38:37.655 INFO:tasks.workunit:Stopping ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'rgw/test_rgw_gc_log.sh'] on client.0... 2019-12-11T16:38:37.656 INFO:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2019-12-11T16:38:37.948 DEBUG:teuthology.parallel:result is None 2019-12-11T16:38:37.949 INFO:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2019-12-11T16:38:38.007 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2019-12-11T16:38:38.007 INFO:teuthology.orchestra.run.smithi079:> rmdir -- /home/ubuntu/cephtest/mnt.0 2019-12-11T16:38:38.098 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2019-12-11T16:38:38.098 INFO:teuthology.run_tasks:Running task ragweed... 2019-12-11T16:38:38.120 DEBUG:tasks.ragweed:ragweed config is {'client.0': {'rgw_server': 'client.0', 'stages': 'prepare,check', 'default-branch': 'ceph-master'}} 2019-12-11T16:38:38.121 INFO:tasks.ragweed:Downloading ragweed... 2019-12-11T16:38:38.121 INFO:tasks.ragweed:Using branch 'wip-abhi-testing-2019-12-10-1759' for ragweed 2019-12-11T16:38:38.122 INFO:teuthology.orchestra.run.smithi079:> git clone -b wip-abhi-testing-2019-12-10-1759 git://git.ceph.com/git/ragweed.git /home/ubuntu/cephtest/ragweed 2019-12-11T16:38:38.184 INFO:teuthology.orchestra.run.smithi079.stderr:fatal: Remote branch wip-abhi-testing-2019-12-10-1759 not found in upstream origin 2019-12-11T16:38:38.185 INFO:teuthology.orchestra.run.smithi079.stdout:Cloning into '/home/ubuntu/cephtest/ragweed'... 2019-12-11T16:38:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 128 2019-12-11T16:38:38.186 INFO:teuthology.orchestra.run.smithi079:> git clone -b ceph-master git://git.ceph.com/git/ragweed.git /home/ubuntu/cephtest/ragweed 2019-12-11T16:38:38.275 INFO:teuthology.orchestra.run.smithi079.stdout:Cloning into '/home/ubuntu/cephtest/ragweed'... 2019-12-11T16:38:38.363 INFO:tasks.ragweed:Creating rgw users... 2019-12-11T16:38:38.364 DEBUG:tasks.ragweed:Creating user sysuser.client.0 on client.0 2019-12-11T16:38:38.364 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid sysuser.client.0 --display-name 'Mr. sysuser.client.0' --access-key RTDAVPNTOYFBNHRTCRLY --secret 4AnXplXTAEe53KIw3rTc/8V3dfEY7mFwKVzB9QHUs5ilJgLPc5jVEg== --email sysuser.client.0+test@test.test --system true 2019-12-11T16:38:38.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.490+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.513+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:38.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.513+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.515 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.522+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:38.515 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.522+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.519 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.526+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:38.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.526+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.534+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:38.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.669+0000 7fb0f7bc7ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:38:38.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.669+0000 7fb0f7bc7ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:38:38.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.669+0000 7fb0f7bc7ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:38:38.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.669+0000 7fb0f7bc7ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:38:38.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.669+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.678+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.678+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.682 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.686+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:38:38.682 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.686+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.693+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:38.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.693+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.701+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:38:38.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.701+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.705+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:38.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.705+0000 7fb0f7bc7ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:38:38.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.705+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.709+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:38.703 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.709+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.720+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:38.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.720+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.731+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:38:38.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.731+0000 7fb0f7bc7ec0 20 zonegroup default 2019-12-11T16:38:38.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.731+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.729 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.736+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:38.729 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.736+0000 7fb0f7bc7ec0 10 Cannot find current period zone using local zone 2019-12-11T16:38:38.729 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.736+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.733 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.740+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:38.734 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.740+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.751+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:38.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.751+0000 7fb0f7bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:38.753 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.759+0000 7fb0f7bc7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:38:38.753 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.759+0000 7fb0f7bc7ec0 20 zone default found 2019-12-11T16:38:38.753 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.759+0000 7fb0f7bc7ec0 20 started sync module instance, tier type = 2019-12-11T16:38:38.753 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:38.759+0000 7fb0f7bc7ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:38:39.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.095+0000 7fb0f7bc7ec0 20 add_watcher() i=0 2019-12-11T16:38:39.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.095+0000 7fb0f7bc7ec0 20 add_watcher() i=1 2019-12-11T16:38:39.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.095+0000 7fb0f7bc7ec0 20 add_watcher() i=2 2019-12-11T16:38:39.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.095+0000 7fb0f7bc7ec0 20 add_watcher() i=3 2019-12-11T16:38:39.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.095+0000 7fb0f7bc7ec0 20 add_watcher() i=4 2019-12-11T16:38:39.090 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.095+0000 7fb0f7bc7ec0 20 add_watcher() i=5 2019-12-11T16:38:39.090 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.095+0000 7fb0f7bc7ec0 20 add_watcher() i=6 2019-12-11T16:38:39.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.135+0000 7fb0f7bc7ec0 20 add_watcher() i=7 2019-12-11T16:38:39.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.135+0000 7fb0f7bc7ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:38:39.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.135+0000 7fb0b57e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:38:39.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.143+0000 7fb0e55f2700 0 -- 172.21.15.79:0/2014978982 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x558a526107e0 msgr2=0x558a52610bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T16:38:39.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.351+0000 7fb0e55f2700 0 --2- 172.21.15.79:0/2014978982 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x558a526107e0 0x558a52610bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-11T16:38:39.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:39.403+0000 7fb0e55f2700 0 -- 172.21.15.79:0/3529419637 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7fb0280224a0 msgr2=0x7fb028024990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:38:40.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.791+0000 7fb0277fe700 20 reqs_thread_entry: start 2019-12-11T16:38:40.786 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.792+0000 7fb0257fa700 20 reqs_thread_entry: start 2019-12-11T16:38:40.786 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.792+0000 7fb0f7bc7ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:38:40.786 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.792+0000 7fb0137fe700 20 reqs_thread_entry: start 2019-12-11T16:38:40.786 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.793+0000 7fb0f7bc7ec0 20 get_system_obj_state: rctx=0x7ffdbbddcf68 obj=default.rgw.meta:users.uid:sysuser.client.0 state=0x558a52696ad0 s->prefetch_data=0 2019-12-11T16:38:40.786 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.793+0000 7fb0f7bc7ec0 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2019-12-11T16:38:40.797 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.804+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x0 2019-12-11T16:38:40.797 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.804+0000 7fb0f7bc7ec0 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-11T16:38:40.797 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.804+0000 7fb0f7bc7ec0 20 get_system_obj_state: rctx=0x7ffdbbddcf78 obj=default.rgw.meta:users.email:sysuser.client.0+test@test.test state=0x558a52698860 s->prefetch_data=0 2019-12-11T16:38:40.798 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.804+0000 7fb0f7bc7ec0 10 cache get: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test : miss 2019-12-11T16:38:40.808 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.815+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x0 2019-12-11T16:38:40.808 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.815+0000 7fb0f7bc7ec0 10 adding default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2019-12-11T16:38:40.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.815+0000 7fb0f7bc7ec0 20 get_system_obj_state: rctx=0x7ffdbbddcf78 obj=default.rgw.meta:users.keys:RTDAVPNTOYFBNHRTCRLY state=0x558a52698860 s->prefetch_data=0 2019-12-11T16:38:40.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.815+0000 7fb0f7bc7ec0 10 cache get: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY : miss 2019-12-11T16:38:40.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.836+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY info.flags=0x0 2019-12-11T16:38:40.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.836+0000 7fb0f7bc7ec0 10 adding default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY to cache LRU end 2019-12-11T16:38:40.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.836+0000 7fb0f7bc7ec0 20 get_system_obj_state: rctx=0x7ffdbbddcb88 obj=default.rgw.meta:users.keys:RTDAVPNTOYFBNHRTCRLY state=0x558a52698860 s->prefetch_data=0 2019-12-11T16:38:40.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.836+0000 7fb0f7bc7ec0 10 cache get: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY : type miss (requested=0x6, cached=0x0) 2019-12-11T16:38:40.841 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.848+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY info.flags=0x0 2019-12-11T16:38:40.841 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.848+0000 7fb0f7bc7ec0 10 moving default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY to cache LRU end 2019-12-11T16:38:40.841 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.848+0000 7fb0f7bc7ec0 20 get_system_obj_state: rctx=0x7ffdbbddcd08 obj=default.rgw.meta:users.keys:RTDAVPNTOYFBNHRTCRLY state=0x558a52698860 s->prefetch_data=0 2019-12-11T16:38:40.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.848+0000 7fb0f7bc7ec0 10 cache get: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY : type miss (requested=0x6, cached=0x0) 2019-12-11T16:38:40.851 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.858+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY info.flags=0x0 2019-12-11T16:38:40.851 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.858+0000 7fb0f7bc7ec0 10 moving default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY to cache LRU end 2019-12-11T16:38:40.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.917+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2019-12-11T16:38:40.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.917+0000 7fb0f7bc7ec0 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-11T16:38:40.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.917+0000 7fb0f7bc7ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=637 2019-12-11T16:38:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.959+0000 7fb0da7fc700 10 RGWWatcher::handle_notify() notify_id 541165879296 cookie 94052575851728 notifier 4695 bl.length()=637 2019-12-11T16:38:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.959+0000 7fb0da7fc700 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2019-12-11T16:38:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:40.959+0000 7fb0da7fc700 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-11T16:38:41.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.260+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2019-12-11T16:38:41.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.260+0000 7fb0f7bc7ec0 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2019-12-11T16:38:41.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.260+0000 7fb0f7bc7ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=202 2019-12-11T16:38:41.292 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.299+0000 7fb0da7fc700 10 RGWWatcher::handle_notify() notify_id 541165879296 cookie 94052575869056 notifier 4695 bl.length()=202 2019-12-11T16:38:41.293 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.299+0000 7fb0da7fc700 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2019-12-11T16:38:41.293 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.299+0000 7fb0da7fc700 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2019-12-11T16:38:41.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.614+0000 7fb0f7bc7ec0 10 cache put: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY info.flags=0x7 2019-12-11T16:38:41.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.614+0000 7fb0f7bc7ec0 10 moving default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY to cache LRU end 2019-12-11T16:38:41.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.614+0000 7fb0f7bc7ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=190 2019-12-11T16:38:41.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.637+0000 7fb0da7fc700 10 RGWWatcher::handle_notify() notify_id 541165879297 cookie 94052575871296 notifier 4695 bl.length()=190 2019-12-11T16:38:41.631 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.637+0000 7fb0da7fc700 10 cache put: name=default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY info.flags=0x7 2019-12-11T16:38:41.631 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.637+0000 7fb0da7fc700 10 moving default.rgw.meta+users.keys+RTDAVPNTOYFBNHRTCRLY to cache LRU end 2019-12-11T16:38:41.716 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T16:38:41.716 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "sysuser.client.0", 2019-12-11T16:38:41.716 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Mr. sysuser.client.0", 2019-12-11T16:38:41.716 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "sysuser.client.0+test@test.test", 2019-12-11T16:38:41.717 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T16:38:41.717 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T16:38:41.717 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [], 2019-12-11T16:38:41.717 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [ 2019-12-11T16:38:41.718 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T16:38:41.718 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "sysuser.client.0", 2019-12-11T16:38:41.718 INFO:teuthology.orchestra.run.smithi079.stdout: "access_key": "RTDAVPNTOYFBNHRTCRLY", 2019-12-11T16:38:41.718 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "4AnXplXTAEe53KIw3rTc/8V3dfEY7mFwKVzB9QHUs5ilJgLPc5jVEg==" 2019-12-11T16:38:41.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T16:38:41.719 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T16:38:41.719 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [], 2019-12-11T16:38:41.719 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [], 2019-12-11T16:38:41.719 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T16:38:41.720 INFO:teuthology.orchestra.run.smithi079.stdout: "system": "true", 2019-12-11T16:38:41.720 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T16:38:41.720 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T16:38:41.720 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T16:38:41.722 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T16:38:41.722 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T16:38:41.722 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T16:38:41.722 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T16:38:41.722 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T16:38:41.722 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T16:38:41.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T16:38:41.723 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T16:38:41.723 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T16:38:41.723 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T16:38:41.723 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T16:38:41.725 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T16:38:41.725 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T16:38:41.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T16:38:41.725 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T16:38:41.725 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T16:38:41.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T16:38:41.725 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T16:38:41.726 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:38:41.793 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.796+0000 7fb0f7bc7ec0 20 remove_watcher() i=0 2019-12-11T16:38:41.794 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.796+0000 7fb0f7bc7ec0 2 removed watcher, disabling cache 2019-12-11T16:38:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.840+0000 7fb0f7bc7ec0 20 remove_watcher() i=1 2019-12-11T16:38:41.875 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.882+0000 7fb0f7bc7ec0 20 remove_watcher() i=2 2019-12-11T16:38:41.948 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:41.955+0000 7fb0f7bc7ec0 20 remove_watcher() i=3 2019-12-11T16:38:42.019 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.025+0000 7fb0f7bc7ec0 20 remove_watcher() i=4 2019-12-11T16:38:42.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.078+0000 7fb0f7bc7ec0 20 remove_watcher() i=5 2019-12-11T16:38:42.138 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.144+0000 7fb0f7bc7ec0 20 remove_watcher() i=6 2019-12-11T16:38:42.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.198+0000 7fb0f7bc7ec0 20 remove_watcher() i=7 2019-12-11T16:38:42.203 DEBUG:tasks.ragweed:Creating user ragweed.client.0 on client.0 2019-12-11T16:38:42.204 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid ragweed.client.0 --display-name 'Mr. ragweed.client.0' --access-key TFEPWBRGOFGYOZCNMHZC --secret FAdgtq6Ebo+yFRIS+Fx/0z6cy3ejpZRlN6p4yen+TOI2NiSIWHytEA== --email ragweed.client.0+test@test.test --system false 2019-12-11T16:38:42.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.365+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.392+0000 7f5a9c694ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.392+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.401+0000 7f5a9c694ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.401+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.406+0000 7f5a9c694ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.406+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.414+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.602+0000 7f5a9c694ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:38:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.602+0000 7f5a9c694ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:38:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.602+0000 7f5a9c694ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:38:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.602+0000 7f5a9c694ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:38:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.602+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.615+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:42.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.615+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.625 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.632+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:38:42.625 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.632+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.641+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:42.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.641+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.665+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:38:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.665+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.671+0000 7f5a9c694ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:42.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.671+0000 7f5a9c694ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:38:42.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.671+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.670 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.677+0000 7f5a9c694ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:42.670 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.677+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.687+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:42.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.687+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.704+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:38:42.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.704+0000 7f5a9c694ec0 20 zonegroup default 2019-12-11T16:38:42.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.704+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.704 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.710+0000 7f5a9c694ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:42.704 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.710+0000 7f5a9c694ec0 10 Cannot find current period zone using local zone 2019-12-11T16:38:42.705 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.710+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.708 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.716+0000 7f5a9c694ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:38:42.709 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.716+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.718 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.725+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:38:42.718 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.725+0000 7f5a9c694ec0 20 rados->read ofs=0 len=0 2019-12-11T16:38:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.742+0000 7f5a9c694ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:38:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.742+0000 7f5a9c694ec0 20 zone default found 2019-12-11T16:38:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.742+0000 7f5a9c694ec0 20 started sync module instance, tier type = 2019-12-11T16:38:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:42.742+0000 7f5a9c694ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:38:43.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.135+0000 7f5a9c694ec0 20 add_watcher() i=0 2019-12-11T16:38:43.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.135+0000 7f5a9c694ec0 20 add_watcher() i=1 2019-12-11T16:38:43.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.135+0000 7f5a9c694ec0 20 add_watcher() i=2 2019-12-11T16:38:43.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.135+0000 7f5a9c694ec0 20 add_watcher() i=3 2019-12-11T16:38:43.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.135+0000 7f5a9c694ec0 20 add_watcher() i=4 2019-12-11T16:38:43.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.135+0000 7f5a9c694ec0 20 add_watcher() i=5 2019-12-11T16:38:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.155+0000 7f5a9c694ec0 20 add_watcher() i=6 2019-12-11T16:38:43.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.168+0000 7f5a9c694ec0 20 add_watcher() i=7 2019-12-11T16:38:43.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.168+0000 7f5a9c694ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:38:43.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.168+0000 7f5a5a7dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:38:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.174+0000 7f5a898be700 0 -- 172.21.15.79:0/2440085910 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x55ab23646e10 msgr2=0x55ab2364d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T16:38:43.369 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.376+0000 7f5a898be700 0 --2- 172.21.15.79:0/2440085910 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x55ab23646e10 0x55ab2364d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-11T16:38:43.463 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:43.470+0000 7f5a898be700 0 -- 172.21.15.79:0/544439734 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7f59cc022470 msgr2=0x7f59cc024960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:38:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.899+0000 7f59d0ff9700 20 reqs_thread_entry: start 2019-12-11T16:38:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.900+0000 7f59ca7fc700 20 reqs_thread_entry: start 2019-12-11T16:38:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.900+0000 7f5a9c694ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:38:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.901+0000 7f59c8ff9700 20 reqs_thread_entry: start 2019-12-11T16:38:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.901+0000 7f5a9c694ec0 20 get_system_obj_state: rctx=0x7ffead57c988 obj=default.rgw.meta:users.uid:ragweed.client.0 state=0x55ab236ca800 s->prefetch_data=0 2019-12-11T16:38:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.901+0000 7f5a9c694ec0 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2019-12-11T16:38:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.907+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x0 2019-12-11T16:38:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.907+0000 7f5a9c694ec0 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-11T16:38:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.907+0000 7f5a9c694ec0 20 get_system_obj_state: rctx=0x7ffead57c998 obj=default.rgw.meta:users.email:ragweed.client.0+test@test.test state=0x55ab236cc590 s->prefetch_data=0 2019-12-11T16:38:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.907+0000 7f5a9c694ec0 10 cache get: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test : miss 2019-12-11T16:38:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.951+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x0 2019-12-11T16:38:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.951+0000 7f5a9c694ec0 10 adding default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2019-12-11T16:38:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.951+0000 7f5a9c694ec0 20 get_system_obj_state: rctx=0x7ffead57c998 obj=default.rgw.meta:users.keys:TFEPWBRGOFGYOZCNMHZC state=0x55ab236cc590 s->prefetch_data=0 2019-12-11T16:38:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.951+0000 7f5a9c694ec0 10 cache get: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC : miss 2019-12-11T16:38:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.959+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC info.flags=0x0 2019-12-11T16:38:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.959+0000 7f5a9c694ec0 10 adding default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC to cache LRU end 2019-12-11T16:38:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.959+0000 7f5a9c694ec0 20 get_system_obj_state: rctx=0x7ffead57c5a8 obj=default.rgw.meta:users.keys:TFEPWBRGOFGYOZCNMHZC state=0x55ab236cc590 s->prefetch_data=0 2019-12-11T16:38:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.959+0000 7f5a9c694ec0 10 cache get: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC : type miss (requested=0x6, cached=0x0) 2019-12-11T16:38:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.963+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC info.flags=0x0 2019-12-11T16:38:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.963+0000 7f5a9c694ec0 10 moving default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC to cache LRU end 2019-12-11T16:38:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.963+0000 7f5a9c694ec0 20 get_system_obj_state: rctx=0x7ffead57c728 obj=default.rgw.meta:users.keys:TFEPWBRGOFGYOZCNMHZC state=0x55ab236cc590 s->prefetch_data=0 2019-12-11T16:38:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.963+0000 7f5a9c694ec0 10 cache get: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC : type miss (requested=0x6, cached=0x0) 2019-12-11T16:38:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.969+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC info.flags=0x0 2019-12-11T16:38:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:44.969+0000 7f5a9c694ec0 10 moving default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC to cache LRU end 2019-12-11T16:38:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.034+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2019-12-11T16:38:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.034+0000 7f5a9c694ec0 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-11T16:38:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.034+0000 7f5a9c694ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=637 2019-12-11T16:38:45.043 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.049+0000 7f5a6affd700 10 RGWWatcher::handle_notify() notify_id 541165879297 cookie 94193521468656 notifier 4707 bl.length()=637 2019-12-11T16:38:45.043 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.050+0000 7f5a6affd700 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2019-12-11T16:38:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.050+0000 7f5a6affd700 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-11T16:38:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.172+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2019-12-11T16:38:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.172+0000 7f5a9c694ec0 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2019-12-11T16:38:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.172+0000 7f5a9c694ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=202 2019-12-11T16:38:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.184+0000 7f5a6affd700 10 RGWWatcher::handle_notify() notify_id 541165879298 cookie 94193521463280 notifier 4707 bl.length()=202 2019-12-11T16:38:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.184+0000 7f5a6affd700 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2019-12-11T16:38:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:45.184+0000 7f5a6affd700 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2019-12-11T16:38:46.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.067+0000 7f5a9c694ec0 10 cache put: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC info.flags=0x7 2019-12-11T16:38:46.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.067+0000 7f5a9c694ec0 10 moving default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC to cache LRU end 2019-12-11T16:38:46.062 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.067+0000 7f5a9c694ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=190 2019-12-11T16:38:46.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.081+0000 7f5a6affd700 10 RGWWatcher::handle_notify() notify_id 541165879299 cookie 94193521463280 notifier 4707 bl.length()=190 2019-12-11T16:38:46.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.081+0000 7f5a6affd700 10 cache put: name=default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC info.flags=0x7 2019-12-11T16:38:46.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.081+0000 7f5a6affd700 10 moving default.rgw.meta+users.keys+TFEPWBRGOFGYOZCNMHZC to cache LRU end 2019-12-11T16:38:46.300 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T16:38:46.300 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "ragweed.client.0", 2019-12-11T16:38:46.300 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Mr. ragweed.client.0", 2019-12-11T16:38:46.301 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "ragweed.client.0+test@test.test", 2019-12-11T16:38:46.301 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T16:38:46.301 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T16:38:46.301 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [], 2019-12-11T16:38:46.301 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [ 2019-12-11T16:38:46.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T16:38:46.301 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "ragweed.client.0", 2019-12-11T16:38:46.303 INFO:teuthology.orchestra.run.smithi079.stdout: "access_key": "TFEPWBRGOFGYOZCNMHZC", 2019-12-11T16:38:46.303 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "FAdgtq6Ebo+yFRIS+Fx/0z6cy3ejpZRlN6p4yen+TOI2NiSIWHytEA==" 2019-12-11T16:38:46.303 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T16:38:46.303 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T16:38:46.304 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [], 2019-12-11T16:38:46.304 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [], 2019-12-11T16:38:46.304 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T16:38:46.304 INFO:teuthology.orchestra.run.smithi079.stdout: "system": "true", 2019-12-11T16:38:46.304 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T16:38:46.304 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T16:38:46.304 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T16:38:46.305 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T16:38:46.305 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T16:38:46.305 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T16:38:46.305 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T16:38:46.305 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T16:38:46.305 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T16:38:46.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T16:38:46.306 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T16:38:46.307 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T16:38:46.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T16:38:46.307 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T16:38:46.307 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:38:46.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.383+0000 7f5a9c694ec0 20 remove_watcher() i=0 2019-12-11T16:38:46.378 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.383+0000 7f5a9c694ec0 2 removed watcher, disabling cache 2019-12-11T16:38:46.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.435+0000 7f5a9c694ec0 20 remove_watcher() i=1 2019-12-11T16:38:46.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.476+0000 7f5a9c694ec0 20 remove_watcher() i=2 2019-12-11T16:38:46.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.532+0000 7f5a9c694ec0 20 remove_watcher() i=3 2019-12-11T16:38:46.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.579+0000 7f5a9c694ec0 20 remove_watcher() i=4 2019-12-11T16:38:46.627 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.633+0000 7f5a9c694ec0 20 remove_watcher() i=5 2019-12-11T16:38:46.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.696+0000 7f5a9c694ec0 20 remove_watcher() i=6 2019-12-11T16:38:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:38:46.761+0000 7f5a9c694ec0 20 remove_watcher() i=7 2019-12-11T16:38:46.769 INFO:tasks.ragweed:Configuring ragweed... 2019-12-11T16:38:46.769 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest/ragweed && ./bootstrap 2019-12-11T16:38:52.859 INFO:teuthology.orchestra.run.smithi079.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2019-12-11T16:38:52.962 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T16:38:53.003 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T16:38:53.003 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T16:38:53.003 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T16:38:53.004 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T16:38:53.009 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T16:38:53.750 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T16:38:54.095 INFO:teuthology.orchestra.run.smithi079.stdout:Package python2-pip-8.1.2-10.el7.noarch already installed and latest version 2019-12-11T16:38:55.009 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T16:38:55.010 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:38:55.010 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libevent-devel.x86_64 0:2.0.21-4.el7 will be installed 2019-12-11T16:38:55.013 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libxml2-devel.x86_64 0:2.9.1-6.el7_2.3 will be installed 2019-12-11T16:38:55.018 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: xz-devel for package: libxml2-devel-2.9.1-6.el7_2.3.x86_64 2019-12-11T16:38:55.071 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libxslt-devel.x86_64 0:1.1.28-5.el7 will be installed 2019-12-11T16:38:55.073 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libgcrypt-devel for package: libxslt-devel-1.1.28-5.el7.x86_64 2019-12-11T16:38:55.077 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed 2019-12-11T16:38:55.080 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:38:55.080 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libgcrypt-devel.x86_64 0:1.5.3-14.el7 will be installed 2019-12-11T16:38:55.081 INFO:teuthology.orchestra.run.smithi079.stdout:--> Processing Dependency: libgpg-error-devel for package: libgcrypt-devel-1.5.3-14.el7.x86_64 2019-12-11T16:38:55.086 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package xz-devel.x86_64 0:5.2.2-1.el7 will be installed 2019-12-11T16:38:55.087 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T16:38:55.087 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libgpg-error-devel.x86_64 0:1.12-3.el7 will be installed 2019-12-11T16:38:55.507 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T16:38:55.741 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:38:55.741 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T16:38:55.743 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:38:55.743 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:38:55.743 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T16:38:55.743 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:38:55.743 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T16:38:55.744 INFO:teuthology.orchestra.run.smithi079.stdout: libevent-devel x86_64 2.0.21-4.el7 base 85 k 2019-12-11T16:38:55.744 INFO:teuthology.orchestra.run.smithi079.stdout: libxml2-devel x86_64 2.9.1-6.el7_2.3 base 1.0 M 2019-12-11T16:38:55.744 INFO:teuthology.orchestra.run.smithi079.stdout: libxslt-devel x86_64 1.1.28-5.el7 base 309 k 2019-12-11T16:38:55.744 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel x86_64 1.2.7-18.el7 base 50 k 2019-12-11T16:38:55.744 INFO:teuthology.orchestra.run.smithi079.stdout:Installing for dependencies: 2019-12-11T16:38:55.744 INFO:teuthology.orchestra.run.smithi079.stdout: libgcrypt-devel x86_64 1.5.3-14.el7 base 129 k 2019-12-11T16:38:55.745 INFO:teuthology.orchestra.run.smithi079.stdout: libgpg-error-devel x86_64 1.12-3.el7 base 16 k 2019-12-11T16:38:55.745 INFO:teuthology.orchestra.run.smithi079.stdout: xz-devel x86_64 5.2.2-1.el7 base 46 k 2019-12-11T16:38:55.745 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:38:55.745 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T16:38:55.745 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T16:38:55.745 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages (+3 Dependent packages) 2019-12-11T16:38:55.746 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:38:55.746 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 1.7 M 2019-12-11T16:38:55.746 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 12 M 2019-12-11T16:38:55.746 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T16:38:55.959 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2019-12-11T16:38:55.960 INFO:teuthology.orchestra.run.smithi079.stdout:Total 7.8 MB/s | 1.7 MB 00:00 2019-12-11T16:38:55.964 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T16:38:55.973 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T16:38:56.022 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T16:38:56.022 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T16:38:56.638 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgpg-error-devel-1.12-3.el7.x86_64 1/7 2019-12-11T16:38:56.964 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgcrypt-devel-1.5.3-14.el7.x86_64 2/7 2019-12-11T16:38:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : zlib-devel-1.2.7-18.el7.x86_64 3/7 2019-12-11T16:38:57.776 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xz-devel-5.2.2-1.el7.x86_64 4/7 2019-12-11T16:38:58.272 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libxml2-devel-2.9.1-6.el7_2.3.x86_64 5/7 2019-12-11T16:38:58.632 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libxslt-devel-1.1.28-5.el7.x86_64 6/7 2019-12-11T16:38:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libevent-devel-2.0.21-4.el7.x86_64 7/7 2019-12-11T16:38:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xz-devel-5.2.2-1.el7.x86_64 1/7 2019-12-11T16:38:59.106 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libxml2-devel-2.9.1-6.el7_2.3.x86_64 2/7 2019-12-11T16:38:59.206 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : zlib-devel-1.2.7-18.el7.x86_64 3/7 2019-12-11T16:38:59.324 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libxslt-devel-1.1.28-5.el7.x86_64 4/7 2019-12-11T16:38:59.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgpg-error-devel-1.12-3.el7.x86_64 5/7 2019-12-11T16:38:59.600 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libevent-devel-2.0.21-4.el7.x86_64 6/7 2019-12-11T16:39:00.129 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgcrypt-devel-1.5.3-14.el7.x86_64 7/7 2019-12-11T16:39:00.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:00.129 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T16:39:00.130 INFO:teuthology.orchestra.run.smithi079.stdout: libevent-devel.x86_64 0:2.0.21-4.el7 libxml2-devel.x86_64 0:2.9.1-6.el7_2.3 2019-12-11T16:39:00.130 INFO:teuthology.orchestra.run.smithi079.stdout: libxslt-devel.x86_64 0:1.1.28-5.el7 zlib-devel.x86_64 0:1.2.7-18.el7 2019-12-11T16:39:00.130 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:00.130 INFO:teuthology.orchestra.run.smithi079.stdout:Dependency Installed: 2019-12-11T16:39:00.130 INFO:teuthology.orchestra.run.smithi079.stdout: libgcrypt-devel.x86_64 0:1.5.3-14.el7 libgpg-error-devel.x86_64 0:1.12-3.el7 2019-12-11T16:39:00.130 INFO:teuthology.orchestra.run.smithi079.stdout: xz-devel.x86_64 0:5.2.2-1.el7 2019-12-11T16:39:00.130 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:00.131 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T16:39:00.307 INFO:teuthology.orchestra.run.smithi079.stdout:New python executable in /home/ubuntu/cephtest/ragweed/virtualenv/bin/python 2019-12-11T16:39:01.983 INFO:teuthology.orchestra.run.smithi079.stdout:Installing setuptools, pip, wheel...done. 2019-12-11T16:39:02.451 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting pip 2019-12-11T16:39:02.559 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB) 2019-12-11T16:39:02.766 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: pip 2019-12-11T16:39:02.766 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: pip 9.0.1 2019-12-11T16:39:02.829 INFO:teuthology.orchestra.run.smithi079.stdout: Uninstalling pip-9.0.1: 2019-12-11T16:39:02.874 INFO:teuthology.orchestra.run.smithi079.stdout: Successfully uninstalled pip-9.0.1 2019-12-11T16:39:03.476 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed pip-19.3.1 2019-12-11T16:39:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T16:39:04.337 INFO:teuthology.orchestra.run.smithi079.stdout:Requirement already satisfied: six in /usr/lib/python2.7/site-packages (1.9.0) 2019-12-11T16:39:05.150 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T16:39:05.392 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting nose 2019-12-11T16:39:05.444 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl (154kB) 2019-12-11T16:39:05.567 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: nose 2019-12-11T16:39:05.653 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed nose-1.3.7 2019-12-11T16:39:06.168 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T16:39:06.729 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting setuptools==32.3.1 2019-12-11T16:39:06.772 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/69/19/b1dff551058ce79d88b1e3688f1c735590d7ddf44d10681512133b35019f/setuptools-32.3.1-py2.py3-none-any.whl (479kB) 2019-12-11T16:39:06.920 INFO:teuthology.orchestra.run.smithi079.stderr:ERROR: kubernetes 10.0.1 has requirement ipaddress>=1.0.17, but you'll have ipaddress 1.0.16 which is incompatible. 2019-12-11T16:39:06.921 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: setuptools 2019-12-11T16:39:06.921 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: setuptools 28.8.0 2019-12-11T16:39:06.937 INFO:teuthology.orchestra.run.smithi079.stdout: Uninstalling setuptools-28.8.0: 2019-12-11T16:39:06.948 INFO:teuthology.orchestra.run.smithi079.stdout: Successfully uninstalled setuptools-28.8.0 2019-12-11T16:39:07.074 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed setuptools-32.3.1 2019-12-11T16:39:07.572 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T16:39:07.882 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting PyYAML 2019-12-11T16:39:07.927 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/8d/c9/e5be955a117a1ac548cdd31e37e8fd7b02ce987f9655f5c7563c656d5dcb/PyYAML-5.2.tar.gz (265kB) 2019-12-11T16:39:08.535 INFO:teuthology.orchestra.run.smithi079.stdout:Requirement already up-to-date: nose>=1.0.0 in ./virtualenv/lib/python2.7/site-packages (from -r requirements.txt (line 2)) (1.3.7) 2019-12-11T16:39:08.649 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting boto!=2.46.0,>=2.6.0 2019-12-11T16:39:08.658 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl (1.4MB) 2019-12-11T16:39:08.993 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting bunch>=1.0.0 2019-12-11T16:39:09.004 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/ef/bf/a4cf1779a4ffb4f610903fa08e15d1f4a8a2f4e3353a02afbe097c5bf4a8/bunch-1.0.1.tar.gz 2019-12-11T16:39:09.661 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting gevent>=1.0 2019-12-11T16:39:09.679 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/e6/e9/3a693414f706e12abe60554cd73c5ae8f848b182ae58018f93d86c9eb418/gevent-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl (5.0MB) 2019-12-11T16:39:10.285 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting httplib2 2019-12-11T16:39:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/ce/2e/87461bfbb7e561203b759b3f7f639e2144226604372830d00a8279960ae1/httplib2-0.14.0.tar.gz (219kB) 2019-12-11T16:39:11.518 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting lxml 2019-12-11T16:39:11.524 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/25/73/730ad7249847f741c7e622f52b971daa5540f6fb87589bc92c717f5aafba/lxml-4.4.2-cp27-cp27mu-manylinux1_x86_64.whl (5.7MB) 2019-12-11T16:39:11.920 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" 2019-12-11T16:39:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/8b/6e/f2d25875713ad0885c8d3c69269697406652e6f64e1a6bd8264f7a609327/greenlet-0.4.15-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-11T16:39:11.993 INFO:teuthology.orchestra.run.smithi079.stdout:Building wheels for collected packages: PyYAML, bunch, httplib2 2019-12-11T16:39:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for PyYAML (setup.py): started 2019-12-11T16:39:12.706 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for PyYAML (setup.py): finished with status 'done' 2019-12-11T16:39:12.707 INFO:teuthology.orchestra.run.smithi079.stdout: Created wheel for PyYAML: filename=PyYAML-5.2-cp27-cp27mu-linux_x86_64.whl size=46299 sha256=8eb9fffac2c5106da30d17f43e8d32c49c9d02a642da14d9668bd37d5268c240 2019-12-11T16:39:12.707 INFO:teuthology.orchestra.run.smithi079.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/54/b7/c7/2ada654ee54483c9329871665aaf4a6056c3ce36f29cf66e67 2019-12-11T16:39:12.736 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for bunch (setup.py): started 2019-12-11T16:39:13.110 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for bunch (setup.py): finished with status 'done' 2019-12-11T16:39:13.110 INFO:teuthology.orchestra.run.smithi079.stdout: Created wheel for bunch: filename=bunch-1.0.1-cp27-none-any.whl size=8347 sha256=b48aa874cd3b988dbf771e72e118de29a92bf2c49f4393918700666b3058aa20 2019-12-11T16:39:13.110 INFO:teuthology.orchestra.run.smithi079.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/56/0f/19/fbbf81e5764e6d8b74501c4357a88c14c94466ec777c03734c 2019-12-11T16:39:13.116 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for httplib2 (setup.py): started 2019-12-11T16:39:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for httplib2 (setup.py): finished with status 'done' 2019-12-11T16:39:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: Created wheel for httplib2: filename=httplib2-0.14.0-cp27-none-any.whl size=96742 sha256=c8ac6c1bc68d43b1b7929cc19ac9cd15123e920c886a66042cd2a24108eeee7f 2019-12-11T16:39:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/51/ab/65/f318b4271019823cf7d7c178ac76a8ea4d6e6271fe9f5ba488 2019-12-11T16:39:13.482 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully built PyYAML bunch httplib2 2019-12-11T16:39:13.597 INFO:teuthology.orchestra.run.smithi079.stderr:ERROR: kubernetes 10.0.1 has requirement ipaddress>=1.0.17, but you'll have ipaddress 1.0.16 which is incompatible. 2019-12-11T16:39:13.598 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: PyYAML, boto, bunch, greenlet, gevent, httplib2, lxml 2019-12-11T16:39:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: PyYAML 3.10 2019-12-11T16:39:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: Not uninstalling pyyaml at /usr/lib64/python2.7/site-packages, outside environment /home/ubuntu/cephtest/ragweed/virtualenv 2019-12-11T16:39:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: Can't uninstall 'PyYAML'. No files were found to uninstall. 2019-12-11T16:39:14.639 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: lxml 3.2.1 2019-12-11T16:39:14.639 INFO:teuthology.orchestra.run.smithi079.stdout: Not uninstalling lxml at /usr/lib64/python2.7/site-packages, outside environment /home/ubuntu/cephtest/ragweed/virtualenv 2019-12-11T16:39:14.640 INFO:teuthology.orchestra.run.smithi079.stdout: Can't uninstall 'lxml'. No files were found to uninstall. 2019-12-11T16:39:14.705 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed PyYAML-5.2 boto-2.49.0 bunch-1.0.1 gevent-1.4.0 greenlet-0.4.15 httplib2-0.14.0 lxml-4.4.2 2019-12-11T16:39:15.062 INFO:teuthology.orchestra.run.smithi079.stdout:running develop 2019-12-11T16:39:15.124 INFO:teuthology.orchestra.run.smithi079.stdout:running egg_info 2019-12-11T16:39:15.125 INFO:teuthology.orchestra.run.smithi079.stdout:creating ragweed.egg-info 2019-12-11T16:39:15.125 INFO:teuthology.orchestra.run.smithi079.stdout:writing requirements to ragweed.egg-info/requires.txt 2019-12-11T16:39:15.125 INFO:teuthology.orchestra.run.smithi079.stdout:writing ragweed.egg-info/PKG-INFO 2019-12-11T16:39:15.125 INFO:teuthology.orchestra.run.smithi079.stdout:writing top-level names to ragweed.egg-info/top_level.txt 2019-12-11T16:39:15.126 INFO:teuthology.orchestra.run.smithi079.stdout:writing dependency_links to ragweed.egg-info/dependency_links.txt 2019-12-11T16:39:15.126 INFO:teuthology.orchestra.run.smithi079.stdout:writing manifest file 'ragweed.egg-info/SOURCES.txt' 2019-12-11T16:39:15.127 INFO:teuthology.orchestra.run.smithi079.stdout:reading manifest file 'ragweed.egg-info/SOURCES.txt' 2019-12-11T16:39:15.127 INFO:teuthology.orchestra.run.smithi079.stdout:writing manifest file 'ragweed.egg-info/SOURCES.txt' 2019-12-11T16:39:15.128 INFO:teuthology.orchestra.run.smithi079.stdout:running build_ext 2019-12-11T16:39:15.128 INFO:teuthology.orchestra.run.smithi079.stdout:Creating /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages/ragweed.egg-link (link to .) 2019-12-11T16:39:15.128 INFO:teuthology.orchestra.run.smithi079.stdout:Adding ragweed 0.0.1 to easy-install.pth file 2019-12-11T16:39:15.128 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.129 INFO:teuthology.orchestra.run.smithi079.stdout:Installed /home/ubuntu/cephtest/ragweed 2019-12-11T16:39:15.130 INFO:teuthology.orchestra.run.smithi079.stdout:Processing dependencies for ragweed==0.0.1 2019-12-11T16:39:15.132 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for isodate>=0.4.4 2019-12-11T16:39:15.132 INFO:teuthology.orchestra.run.smithi079.stdout:Reading https://pypi.python.org/simple/isodate/ 2019-12-11T16:39:15.223 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading https://files.pythonhosted.org/packages/b1/80/fb8c13a4cd38eb5021dc3741a9e588e4d1de88d895c1910c6fc8a08b7a70/isodate-0.6.0.tar.gz#sha256=2e364a3d5759479cdb2d37cce6b9376ea504db2ff90252a2e5b7cc89cc9ff2d8 2019-12-11T16:39:15.273 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: isodate 0.6.0 2019-12-11T16:39:15.274 INFO:teuthology.orchestra.run.smithi079.stdout:Processing isodate-0.6.0.tar.gz 2019-12-11T16:39:15.280 INFO:teuthology.orchestra.run.smithi079.stdout:Writing /tmp/easy_install-GpMZVT/isodate-0.6.0/setup.cfg 2019-12-11T16:39:15.281 INFO:teuthology.orchestra.run.smithi079.stdout:Running isodate-0.6.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-GpMZVT/isodate-0.6.0/egg-dist-tmp-N5EwNe 2019-12-11T16:39:15.547 INFO:teuthology.orchestra.run.smithi079.stderr:zip_safe flag not set; analyzing archive contents... 2019-12-11T16:39:15.562 INFO:teuthology.orchestra.run.smithi079.stdout:Moving isodate-0.6.0-py2.7.egg to /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-11T16:39:15.563 INFO:teuthology.orchestra.run.smithi079.stdout:Adding isodate 0.6.0 to easy-install.pth file 2019-12-11T16:39:15.564 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.565 INFO:teuthology.orchestra.run.smithi079.stdout:Installed /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages/isodate-0.6.0-py2.7.egg 2019-12-11T16:39:15.587 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for gevent==1.4.0 2019-12-11T16:39:15.588 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: gevent 1.4.0 2019-12-11T16:39:15.588 INFO:teuthology.orchestra.run.smithi079.stdout:Adding gevent 1.4.0 to easy-install.pth file 2019-12-11T16:39:15.589 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.589 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-11T16:39:15.589 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for bunch==1.0.1 2019-12-11T16:39:15.589 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: bunch 1.0.1 2019-12-11T16:39:15.589 INFO:teuthology.orchestra.run.smithi079.stdout:Adding bunch 1.0.1 to easy-install.pth file 2019-12-11T16:39:15.590 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.590 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-11T16:39:15.592 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for PyYAML==5.2 2019-12-11T16:39:15.592 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: PyYAML 5.2 2019-12-11T16:39:15.592 INFO:teuthology.orchestra.run.smithi079.stdout:Adding PyYAML 5.2 to easy-install.pth file 2019-12-11T16:39:15.593 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.593 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-11T16:39:15.593 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for boto==2.49.0 2019-12-11T16:39:15.593 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: boto 2.49.0 2019-12-11T16:39:15.593 INFO:teuthology.orchestra.run.smithi079.stdout:Adding boto 2.49.0 to easy-install.pth file 2019-12-11T16:39:15.593 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.594 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-11T16:39:15.594 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for six==1.9.0 2019-12-11T16:39:15.594 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: six 1.9.0 2019-12-11T16:39:15.594 INFO:teuthology.orchestra.run.smithi079.stdout:Adding six 1.9.0 to easy-install.pth file 2019-12-11T16:39:15.594 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.594 INFO:teuthology.orchestra.run.smithi079.stdout:Using /usr/lib/python2.7/site-packages 2019-12-11T16:39:15.603 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for greenlet==0.4.15 2019-12-11T16:39:15.603 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: greenlet 0.4.15 2019-12-11T16:39:15.603 INFO:teuthology.orchestra.run.smithi079.stdout:Adding greenlet 0.4.15 to easy-install.pth file 2019-12-11T16:39:15.604 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:39:15.604 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-11T16:39:15.604 INFO:teuthology.orchestra.run.smithi079.stdout:Finished processing dependencies for ragweed==0.0.1 2019-12-11T16:39:15.622 INFO:teuthology.orchestra.run.smithi079:> cat > /home/ubuntu/cephtest/archive/ragweed.client.0.conf 2019-12-11T16:39:15.674 INFO:tasks.ragweed:Configuring boto... 2019-12-11T16:39:15.674 INFO:teuthology.orchestra.run.smithi079:> cat > /home/ubuntu/cephtest/boto.cfg 2019-12-11T16:39:15.764 INFO:teuthology.orchestra.run.smithi079:ragweed tests against rgw> RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw' 2019-12-11T16:39:49.878 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_multipart_defer_complete.test ... ok 2019-12-11T16:40:09.814 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_multipart_defer_update_complete.test ... ok 2019-12-11T16:40:32.306 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_multipart_index_versioning.test ... ok 2019-12-11T16:40:58.065 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_multipart_simple.test ... ok 2019-12-11T16:41:21.353 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_obj_storage_class.test ... ok 2019-12-11T16:41:29.952 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:09:16.897 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.034 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:09:16.979 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.051 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:09:16.996 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.116 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:09:17.061 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.166 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:09:17.078 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.259 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:09:17.172 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.745 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:09:17.690 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.749 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:09:17.694 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:41:30.862 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:09:17.776 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:42:21.591 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_obj_storage_class_copy.test ... ok 2019-12-11T16:43:10.038 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_obj_storage_class_multipart.test ... ok 2019-12-11T16:43:36.313 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:11:23.260 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.322 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:11:23.270 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.330 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:11:23.278 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.334 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:11:23.281 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.350 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:11:23.265 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.369 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:11:23.284 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.428 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:11:23.375 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.439 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:11:23.386 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:43:36.518 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:11:23.433 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:44:55.288 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_obj_storage_class_multipart_copy.test ... ok 2019-12-11T16:45:24.148 INFO:teuthology.orchestra.run.smithi079.stderr:ragweed.tests.tests.r_test_small_obj_data.test ... ok 2019-12-11T16:45:24.150 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T16:45:24.150 INFO:teuthology.orchestra.run.smithi079.stderr:---------------------------------------------------------------------- 2019-12-11T16:45:24.150 INFO:teuthology.orchestra.run.smithi079.stderr:Ran 9 tests in 367.955s 2019-12-11T16:45:24.150 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T16:45:24.150 INFO:teuthology.orchestra.run.smithi079.stderr:OK 2019-12-11T16:45:24.174 INFO:tasks.ragweed:Cleaning up boto... 2019-12-11T16:45:24.174 INFO:teuthology.orchestra.run.smithi079:> rm /home/ubuntu/cephtest/boto.cfg 2019-12-11T16:45:24.241 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid sysuser.client.0 --purge-data 2019-12-11T16:45:24.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.469+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.492+0000 7fd85a990ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:24.477 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.492+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.501+0000 7fd85a990ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:24.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.501+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.506+0000 7fd85a990ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:24.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.506+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.498 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.514+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:24.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.695+0000 7fd85a990ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:45:24.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.695+0000 7fd85a990ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:45:24.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.695+0000 7fd85a990ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:45:24.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.695+0000 7fd85a990ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:45:24.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.696+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.703+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:24.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.703+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.704 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.720+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:45:24.705 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.720+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.729+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:24.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.729+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.741+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:45:24.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.741+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.730 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.746+0000 7fd85a990ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:24.730 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.746+0000 7fd85a990ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:45:24.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.746+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.751+0000 7fd85a990ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:24.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.751+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.759+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:24.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.759+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.775+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:45:24.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.775+0000 7fd85a990ec0 20 zonegroup default 2019-12-11T16:45:24.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.775+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.779+0000 7fd85a990ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:24.764 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.779+0000 7fd85a990ec0 10 Cannot find current period zone using local zone 2019-12-11T16:45:24.764 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.779+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.768 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.784+0000 7fd85a990ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:24.768 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.784+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.774 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.790+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.790+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.803+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:45:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.803+0000 7fd85a990ec0 20 zone default found 2019-12-11T16:45:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.803+0000 7fd85a990ec0 20 started sync module instance, tier type = 2019-12-11T16:45:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:24.803+0000 7fd85a990ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:45:25.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.199+0000 7fd85a990ec0 20 add_watcher() i=0 2019-12-11T16:45:25.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.199+0000 7fd85a990ec0 20 add_watcher() i=1 2019-12-11T16:45:25.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.199+0000 7fd85a990ec0 20 add_watcher() i=2 2019-12-11T16:45:25.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.199+0000 7fd85a990ec0 20 add_watcher() i=3 2019-12-11T16:45:25.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.199+0000 7fd85a990ec0 20 add_watcher() i=4 2019-12-11T16:45:25.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.199+0000 7fd85a990ec0 20 add_watcher() i=5 2019-12-11T16:45:25.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.199+0000 7fd85a990ec0 20 add_watcher() i=6 2019-12-11T16:45:25.216 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.231+0000 7fd85a990ec0 20 add_watcher() i=7 2019-12-11T16:45:25.216 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.231+0000 7fd85a990ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:45:25.216 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.231+0000 7fd79bfff700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:45:25.218 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.234+0000 7fd8473b9700 0 -- 172.21.15.79:0/2453537725 >> v1:172.21.15.79:6789/0 conn(0x55d511565840 legacy=0x55d511567d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2019-12-11T16:45:25.261 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:25.277+0000 7fd8473b9700 0 -- 172.21.15.79:0/1990838850 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x55d511577ce0 msgr2=0x55d51157a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:45:26.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.006+0000 7fd7867fc700 20 reqs_thread_entry: start 2019-12-11T16:45:26.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.007+0000 7fd77ffff700 20 reqs_thread_entry: start 2019-12-11T16:45:26.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.007+0000 7fd85a990ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:45:26.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.007+0000 7fd77e7fc700 20 reqs_thread_entry: start 2019-12-11T16:45:26.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.007+0000 7fd85a990ec0 20 get_system_obj_state: rctx=0x7ffcf528a098 obj=default.rgw.meta:users.uid:sysuser.client.0 state=0x55d5115e4730 s->prefetch_data=0 2019-12-11T16:45:26.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.007+0000 7fd85a990ec0 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2019-12-11T16:45:27.012 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.028+0000 7fd85a990ec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x16 2019-12-11T16:45:27.012 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.028+0000 7fd85a990ec0 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-11T16:45:27.012 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.028+0000 7fd85a990ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:27.013 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.028+0000 7fd85a990ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:27.013 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.028+0000 7fd85a990ec0 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T16:45:27.013 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.028+0000 7fd85a990ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:27.024 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.040+0000 7fd85a990ec0 20 rados_obj.operate() r=0 bl.length=448 2019-12-11T16:45:27.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.040+0000 7fd85a990ec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x11 2019-12-11T16:45:27.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.040+0000 7fd85a990ec0 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-11T16:45:27.037 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.053+0000 7fd85a990ec0 10 removing key index: RTDAVPNTOYFBNHRTCRLY 2019-12-11T16:45:27.037 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.053+0000 7fd85a990ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=170 2019-12-11T16:45:27.045 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.061+0000 7fd82d7fa700 10 RGWWatcher::handle_notify() notify_id 579820584964 cookie 94373607174368 notifier 4466 bl.length()=170 2019-12-11T16:45:27.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.221+0000 7fd85a990ec0 10 removing email index: sysuser.client.0+test@test.test 2019-12-11T16:45:27.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.221+0000 7fd85a990ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=182 2019-12-11T16:45:27.216 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.232+0000 7fd82d7fa700 10 RGWWatcher::handle_notify() notify_id 579820584962 cookie 94373607158880 notifier 4466 bl.length()=182 2019-12-11T16:45:27.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.362+0000 7fd85a990ec0 10 removing user buckets index 2019-12-11T16:45:27.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.362+0000 7fd85a990ec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=173 2019-12-11T16:45:27.390 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:27.406+0000 7fd82d7fa700 10 RGWWatcher::handle_notify() notify_id 579820584960 cookie 94373607155696 notifier 4466 bl.length()=173 2019-12-11T16:45:28.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:28.087+0000 7fd85a990ec0 10 removing user index: sysuser.client.0 2019-12-11T16:45:28.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:28.087+0000 7fd85a990ec0 10 removing default.rgw.meta+users.uid+sysuser.client.0 from cache 2019-12-11T16:45:28.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:28.087+0000 7fd85a990ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=165 2019-12-11T16:45:28.079 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:28.094+0000 7fd82d7fa700 10 RGWWatcher::handle_notify() notify_id 579820584965 cookie 94373607153664 notifier 4466 bl.length()=165 2019-12-11T16:45:28.896 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:28.912+0000 7fd85a990ec0 20 remove_watcher() i=0 2019-12-11T16:45:28.897 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:28.912+0000 7fd85a990ec0 2 removed watcher, disabling cache 2019-12-11T16:45:28.943 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:28.959+0000 7fd85a990ec0 20 remove_watcher() i=1 2019-12-11T16:45:28.989 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.005+0000 7fd85a990ec0 20 remove_watcher() i=2 2019-12-11T16:45:29.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.063+0000 7fd85a990ec0 20 remove_watcher() i=3 2019-12-11T16:45:29.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.101+0000 7fd85a990ec0 20 remove_watcher() i=4 2019-12-11T16:45:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.140+0000 7fd85a990ec0 20 remove_watcher() i=5 2019-12-11T16:45:29.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.174+0000 7fd85a990ec0 20 remove_watcher() i=6 2019-12-11T16:45:29.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.205+0000 7fd85a990ec0 20 remove_watcher() i=7 2019-12-11T16:45:29.198 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid ragweed.client.0 --purge-data 2019-12-11T16:45:29.314 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.330+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.351+0000 7f7fc3b64ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.351+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.361 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.377+0000 7f7fc3b64ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:29.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.377+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.382+0000 7f7fc3b64ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.382+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.389+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:29.570 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.586+0000 7f7fc3b64ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:45:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.586+0000 7f7fc3b64ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:45:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.586+0000 7f7fc3b64ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:45:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.586+0000 7f7fc3b64ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:45:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.586+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.581 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.597+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:29.581 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.597+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.598 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.613+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:45:29.598 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.614+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.625+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:29.610 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.625+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.645+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:45:29.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.645+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.633 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.649+0000 7f7fc3b64ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:29.633 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.649+0000 7f7fc3b64ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:45:29.633 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.649+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.638 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.654+0000 7f7fc3b64ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:29.638 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.654+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.663+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:29.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.663+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.680+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:45:29.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.680+0000 7f7fc3b64ec0 20 zonegroup default 2019-12-11T16:45:29.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.680+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.668 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.685+0000 7f7fc3b64ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:29.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.685+0000 7f7fc3b64ec0 10 Cannot find current period zone using local zone 2019-12-11T16:45:29.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.685+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.674 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.689+0000 7f7fc3b64ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:45:29.674 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.689+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.682 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.698+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:45:29.682 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.698+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:29.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.712+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:45:29.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.712+0000 7f7fc3b64ec0 20 zone default found 2019-12-11T16:45:29.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.712+0000 7f7fc3b64ec0 20 started sync module instance, tier type = 2019-12-11T16:45:29.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:29.712+0000 7f7fc3b64ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:45:30.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.124+0000 7f7fc3b64ec0 20 add_watcher() i=0 2019-12-11T16:45:30.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.124+0000 7f7fc3b64ec0 20 add_watcher() i=1 2019-12-11T16:45:30.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.124+0000 7f7fc3b64ec0 20 add_watcher() i=2 2019-12-11T16:45:30.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.124+0000 7f7fc3b64ec0 20 add_watcher() i=3 2019-12-11T16:45:30.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.124+0000 7f7fc3b64ec0 20 add_watcher() i=4 2019-12-11T16:45:30.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.124+0000 7f7fc3b64ec0 20 add_watcher() i=5 2019-12-11T16:45:30.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.147+0000 7f7fc3b64ec0 20 add_watcher() i=6 2019-12-11T16:45:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.154+0000 7f7fc3b64ec0 20 add_watcher() i=7 2019-12-11T16:45:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.154+0000 7f7fc3b64ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:45:30.139 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.154+0000 7f7f817e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:45:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.173+0000 7f7fb058d700 0 -- 172.21.15.79:0/325729402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562aafcc2c30 msgr2=0x562aafcc5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:45:30.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:30.213+0000 7f7fb158f700 0 -- 172.21.15.79:0/246296344 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7f7ef4022490 msgr2=0x7f7ef4024960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:45:31.565 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.581+0000 7f7ef37fe700 20 reqs_thread_entry: start 2019-12-11T16:45:31.565 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.581+0000 7f7ef17fa700 20 reqs_thread_entry: start 2019-12-11T16:45:31.565 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.581+0000 7f7fc3b64ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:45:31.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.582+0000 7f7edf7fe700 20 reqs_thread_entry: start 2019-12-11T16:45:31.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.582+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea828 obj=default.rgw.meta:users.uid:ragweed.client.0 state=0x562aafd40860 s->prefetch_data=0 2019-12-11T16:45:31.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.582+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2019-12-11T16:45:31.584 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.600+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x16 2019-12-11T16:45:31.584 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.600+0000 7f7fc3b64ec0 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-11T16:45:31.584 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.600+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:31.584 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.600+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:31.585 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.600+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T16:45:31.585 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.600+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:31.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.613+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=448 2019-12-11T16:45:31.597 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.613+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x11 2019-12-11T16:45:31.597 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.613+0000 7f7fc3b64ec0 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-11T16:45:31.619 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.635+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-conf state=0x562aafd42180 s->prefetch_data=0 2019-12-11T16:45:31.619 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.635+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : miss 2019-12-11T16:45:31.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.645+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x6 2019-12-11T16:45:31.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.645+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-conf to cache LRU end 2019-12-11T16:45:31.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.645+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:31.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.645+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:31.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.645+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:31.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.645+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:31.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.658+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=200 2019-12-11T16:45:31.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.658+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x1 2019-12-11T16:45:31.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.658+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-conf to cache LRU end 2019-12-11T16:45:31.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.658+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 state=0x562aafd44750 s->prefetch_data=0 2019-12-11T16:45:31.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.658+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 : miss 2019-12-11T16:45:31.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 info.flags=0x16 2019-12-11T16:45:31.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 to cache LRU end 2019-12-11T16:45:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.679+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:31.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.716+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=292 2019-12-11T16:45:31.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.716+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 info.flags=0x13 2019-12-11T16:45:31.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.716+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 to cache LRU end 2019-12-11T16:45:31.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.716+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:31.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.716+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:31.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.716+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:31.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.741+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-conf[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1]) start [] num_entries 1100 2019-12-11T16:45:31.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_defer_complete[] 2019-12-11T16:45:31.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_defer_update_complete[] 2019-12-11T16:45:31.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_index_versioning[] 2019-12-11T16:45:31.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_simple[] 2019-12-11T16:45:31.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class[] 2019-12-11T16:45:31.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class_copy[] 2019-12-11T16:45:31.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart[] 2019-12-11T16:45:31.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart_copy[] 2019-12-11T16:45:31.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_small_obj_data[] 2019-12-11T16:45:31.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.773+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:31.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.798+0000 7f7fc3b64ec0 10 manifest: total_size = 5842 2019-12-11T16:45:31.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.798+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.13 2019-12-11T16:45:31.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.798+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:31.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.798+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:31.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.798+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:31.951 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.967+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:31.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.977+0000 7f7fc3b64ec0 10 manifest: total_size = 9911 2019-12-11T16:45:31.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.977+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.30 2019-12-11T16:45:31.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.977+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:31.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.977+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:31.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:31.977+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:32.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.144+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.139 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.155+0000 7f7fc3b64ec0 10 manifest: total_size = 9698 2019-12-11T16:45:32.139 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.155+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.56 2019-12-11T16:45:32.140 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.155+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.140 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.155+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:32.140 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.155+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:32.319 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.335+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.358+0000 7f7fc3b64ec0 10 manifest: total_size = 5810 2019-12-11T16:45:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.358+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.72 2019-12-11T16:45:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.358+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.358+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.358+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:32.467 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.483+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.494+0000 7f7fc3b64ec0 10 manifest: total_size = 3888 2019-12-11T16:45:32.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.494+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.87 2019-12-11T16:45:32.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.494+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.494+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:32.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.494+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.651+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.659+0000 7f7fc3b64ec0 10 manifest: total_size = 12925 2019-12-11T16:45:32.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.659+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.120 2019-12-11T16:45:32.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.659+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.659+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:32.644 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.659+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:32.857 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.873+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.910+0000 7f7fc3b64ec0 10 manifest: total_size = 6132 2019-12-11T16:45:32.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.910+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.189 2019-12-11T16:45:32.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.910+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:32.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.910+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:32.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:32.910+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:33.077 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.093+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:33.094 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.110+0000 7f7fc3b64ec0 10 manifest: total_size = 6119 2019-12-11T16:45:33.094 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.110+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.246 2019-12-11T16:45:33.094 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.110+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:33.095 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.110+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:33.095 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.110+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:33.301 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.317+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:33.314 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.330+0000 7f7fc3b64ec0 10 manifest: total_size = 6762 2019-12-11T16:45:33.314 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.330+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.322 2019-12-11T16:45:33.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.330+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x562aafd49938 s->prefetch_data=0 2019-12-11T16:45:33.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.330+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd49938 2019-12-11T16:45:33.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.330+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 2019-12-11T16:45:33.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.545+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-conf[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1]) start _multipart_[] num_entries 1001 2019-12-11T16:45:33.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.545+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:33.582 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.598+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-conf[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:33.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.646+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-conf state=0x562aafd4a980 s->prefetch_data=0 2019-12-11T16:45:33.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.646+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:33.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.656+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x16 2019-12-11T16:45:33.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.656+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-conf to cache LRU end 2019-12-11T16:45:33.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.656+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:33.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.656+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:33.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.656+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:33.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.656+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-conf from cache 2019-12-11T16:45:33.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.656+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=162 2019-12-11T16:45:33.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.685+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584960 cookie 94741337885680 notifier 4475 bl.length()=162 2019-12-11T16:45:33.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.845+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-conf:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.1 from cache 2019-12-11T16:45:33.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.845+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=219 2019-12-11T16:45:33.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:33.861+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584966 cookie 94741337911440 notifier 4475 bl.length()=219 2019-12-11T16:45:34.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.545+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-complete.1 state=0x562aafcab600 s->prefetch_data=0 2019-12-11T16:45:34.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.545+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : miss 2019-12-11T16:45:34.545 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.561+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x6 2019-12-11T16:45:34.545 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.561+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2019-12-11T16:45:34.545 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.561+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:34.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.561+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:34.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.561+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:34.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.561+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.571+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=229 2019-12-11T16:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.571+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x1 2019-12-11T16:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.571+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2019-12-11T16:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.571+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 state=0x562aafd490b0 s->prefetch_data=0 2019-12-11T16:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.571+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 : miss 2019-12-11T16:45:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 info.flags=0x16 2019-12-11T16:45:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 to cache LRU end 2019-12-11T16:45:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.591+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:34.607 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.623+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=321 2019-12-11T16:45:34.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.623+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 info.flags=0x13 2019-12-11T16:45:34.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.623+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 to cache LRU end 2019-12-11T16:45:34.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.623+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:34.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.623+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:34.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.623+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 2019-12-11T16:45:34.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.639+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-defer-complete.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2]) start [] num_entries 1100 2019-12-11T16:45:34.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.655+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo[] 2019-12-11T16:45:34.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.655+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:34.654 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.670+0000 7f7fc3b64ec0 10 manifest: total_size = 26081 2019-12-11T16:45:34.655 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.670+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.20 2019-12-11T16:45:34.655 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.670+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:34.655 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.670+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:34.655 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.670+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 2019-12-11T16:45:34.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=7672 rules.size()=3 2019-12-11T16:45:34.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=7672 2019-12-11T16:45:34.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=7672 stripe_ofs=7672 part_ofs=7672 rule->part_size=7672 2019-12-11T16:45:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=7672 rules.size()=3 2019-12-11T16:45:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4197880 part_ofs=7672 rule->part_size=7672 2019-12-11T16:45:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=15344 stripe_ofs=15344 part_ofs=15344 rule->part_size=7670 2019-12-11T16:45:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=7670 rules.size()=3 2019-12-11T16:45:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4205552 part_ofs=15344 rule->part_size=7670 2019-12-11T16:45:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=23014 stripe_ofs=23014 part_ofs=23014 rule->part_size=3067 2019-12-11T16:45:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=3067 rules.size()=3 2019-12-11T16:45:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4213222 part_ofs=23014 rule->part_size=3067 2019-12-11T16:45:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=26081 stripe_ofs=26081 part_ofs=26081 rule->part_size=3067 2019-12-11T16:45:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.838+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.20 2019-12-11T16:45:34.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.927+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-defer-complete.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2]) start _multipart_[] num_entries 1001 2019-12-11T16:45:34.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.927+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:34.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:34.937+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-complete.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:34.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.002+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-complete.1 state=0x562aafd4a750 s->prefetch_data=0 2019-12-11T16:45:34.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.002+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:35.001 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.017+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x16 2019-12-11T16:45:35.001 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.017+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2019-12-11T16:45:35.001 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.017+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:35.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.017+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:35.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.017+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:35.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.017+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 from cache 2019-12-11T16:45:35.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.017+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=191 2019-12-11T16:45:35.010 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.027+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584961 cookie 94741337887792 notifier 4475 bl.length()=191 2019-12-11T16:45:35.474 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.490+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2 from cache 2019-12-11T16:45:35.474 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.490+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=248 2019-12-11T16:45:35.492 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:35.508+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584961 cookie 94741337885680 notifier 4475 bl.length()=248 2019-12-11T16:45:35.993 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.005+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-update-complete.1 state=0x562aafd4aa00 s->prefetch_data=0 2019-12-11T16:45:35.993 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.005+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : miss 2019-12-11T16:45:36.006 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.022+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x6 2019-12-11T16:45:36.016 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.022+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2019-12-11T16:45:36.016 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.022+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:36.016 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.022+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:36.016 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.022+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:36.017 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.022+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:36.020 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.037+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=236 2019-12-11T16:45:36.021 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.037+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x1 2019-12-11T16:45:36.021 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.037+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2019-12-11T16:45:36.021 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.037+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 state=0x562aafd4a750 s->prefetch_data=0 2019-12-11T16:45:36.021 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.037+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 : miss 2019-12-11T16:45:36.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 info.flags=0x16 2019-12-11T16:45:36.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 to cache LRU end 2019-12-11T16:45:36.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:36.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:36.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:36.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:36.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:36.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:36.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.062+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:36.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.084+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=328 2019-12-11T16:45:36.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.084+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 info.flags=0x13 2019-12-11T16:45:36.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.084+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 to cache LRU end 2019-12-11T16:45:36.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.084+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:36.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.084+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:36.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.084+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 2019-12-11T16:45:36.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.105+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-defer-update-complete.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3]) start [] num_entries 1100 2019-12-11T16:45:36.105 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.121+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo[] 2019-12-11T16:45:36.105 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.121+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:36.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.135+0000 7f7fc3b64ec0 10 manifest: total_size = 1848 2019-12-11T16:45:36.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.135+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.40 2019-12-11T16:45:36.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.135+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:36.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.135+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:36.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.136+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 2019-12-11T16:45:36.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=515 rules.size()=2 2019-12-11T16:45:36.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=515 2019-12-11T16:45:36.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=515 stripe_ofs=515 part_ofs=515 rule->part_size=515 2019-12-11T16:45:36.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=515 rules.size()=2 2019-12-11T16:45:36.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190723 part_ofs=515 rule->part_size=515 2019-12-11T16:45:36.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=1030 stripe_ofs=1030 part_ofs=1030 rule->part_size=515 2019-12-11T16:45:36.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=515 rules.size()=2 2019-12-11T16:45:36.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4191238 part_ofs=1030 rule->part_size=515 2019-12-11T16:45:36.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=1545 stripe_ofs=1545 part_ofs=1545 rule->part_size=303 2019-12-11T16:45:36.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=303 rules.size()=2 2019-12-11T16:45:36.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4191753 part_ofs=1545 rule->part_size=303 2019-12-11T16:45:36.254 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=1848 stripe_ofs=1848 part_ofs=1848 rule->part_size=303 2019-12-11T16:45:36.254 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.268+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.40 2019-12-11T16:45:36.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.410+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-defer-update-complete.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3]) start _multipart_[] num_entries 1001 2019-12-11T16:45:36.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.410+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.420+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-update-complete.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:36.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.488+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-update-complete.1 state=0x562aafd47b10 s->prefetch_data=0 2019-12-11T16:45:36.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.488+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:36.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.500+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x16 2019-12-11T16:45:36.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.500+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2019-12-11T16:45:36.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.500+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:36.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.500+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:36.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.500+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:36.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.500+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 from cache 2019-12-11T16:45:36.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.500+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=198 2019-12-11T16:45:36.496 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.512+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584967 cookie 94741337890976 notifier 4475 bl.length()=198 2019-12-11T16:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.740+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3 from cache 2019-12-11T16:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.740+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=255 2019-12-11T16:45:36.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:36.782+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584960 cookie 94741337929168 notifier 4475 bl.length()=255 2019-12-11T16:45:37.057 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.073+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-index-versioning.1 state=0x562aafd4a750 s->prefetch_data=0 2019-12-11T16:45:37.057 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.073+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : miss 2019-12-11T16:45:37.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.087+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x6 2019-12-11T16:45:37.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.087+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2019-12-11T16:45:37.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.087+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:37.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.087+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:37.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.087+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:37.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.087+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.100+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=231 2019-12-11T16:45:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.100+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x1 2019-12-11T16:45:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.100+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2019-12-11T16:45:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.100+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 state=0x562aafd490b0 s->prefetch_data=0 2019-12-11T16:45:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.100+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 : miss 2019-12-11T16:45:37.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 info.flags=0x16 2019-12-11T16:45:37.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 to cache LRU end 2019-12-11T16:45:37.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:37.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:37.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:37.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:37.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:37.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:37.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.123+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:37.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.141+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=323 2019-12-11T16:45:37.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.141+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 info.flags=0x13 2019-12-11T16:45:37.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.141+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 to cache LRU end 2019-12-11T16:45:37.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.141+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:37.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.141+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:37.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.141+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.139 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.156+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-index-versioning.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4]) start [] num_entries 1100 2019-12-11T16:45:37.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.191+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo[b-3-cMapfdH0pCJjjoyRUZqVSjDvRW-] 2019-12-11T16:45:37.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.191+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.216 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.232+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:37.239 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.251+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 20r9p5xykf40ioqy0idxi83eard4tlrm 2019-12-11T16:45:37.239 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.251+0000 7f7fc3b64ec0 20 get_obj_state_impl: setting s->olh_tag to xy1vej7z4nlvkbka9jsndv0zkl95f2da 2019-12-11T16:45:37.239 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.251+0000 7f7fc3b64ec0 20 bucket_index_guard_olh_op(): olh_state.olh_tag=xy1vej7z4nlvkbka9jsndv0zkl95f2da 2019-12-11T16:45:37.323 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.339+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.399+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.399+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.412+0000 7f7fc3b64ec0 20 olh_log_entry: epoch=3 op=2 key=[] 2019-12-11T16:45:37.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.412+0000 7f7fc3b64ec0 20 olh_log_entry: epoch=3 op=3 key=foo[b-3-cMapfdH0pCJjjoyRUZqVSjDvRW-] 2019-12-11T16:45:37.397 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.412+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-index-versioning.1:_:b-3-cMapfdH0pCJjjoyRUZqVSjDvRW-_foo state=0x562aafd4cda8 s->prefetch_data=0 2019-12-11T16:45:37.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.432+0000 7f7fc3b64ec0 10 manifest: total_size = 74198 2019-12-11T16:45:37.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.432+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.55 2019-12-11T16:45:37.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.432+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-index-versioning.1:_:b-3-cMapfdH0pCJjjoyRUZqVSjDvRW-_foo state=0x562aafd4cda8 s->prefetch_data=0 2019-12-11T16:45:37.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.432+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cda8 2019-12-11T16:45:37.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.432+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.530 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2019-12-11T16:45:37.530 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=20613 2019-12-11T16:45:37.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2019-12-11T16:45:37.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2019-12-11T16:45:37.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4210821 part_ofs=20613 rule->part_size=20613 2019-12-11T16:45:37.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=20613 2019-12-11T16:45:37.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2019-12-11T16:45:37.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4231434 part_ofs=41226 rule->part_size=20613 2019-12-11T16:45:37.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=61839 stripe_ofs=61839 part_ofs=61839 rule->part_size=12359 2019-12-11T16:45:37.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=12359 rules.size()=2 2019-12-11T16:45:37.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4252047 part_ofs=61839 rule->part_size=12359 2019-12-11T16:45:37.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=74198 stripe_ofs=74198 part_ofs=74198 rule->part_size=12359 2019-12-11T16:45:37.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.546+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.15tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.55 2019-12-11T16:45:37.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.682+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.682+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.776+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.827+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 2019-12-11T16:45:37.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.827+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-index-versioning.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4]) start _multipart_[] num_entries 1001 2019-12-11T16:45:37.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.827+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:37.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.839+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-index-versioning.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:37.906 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.922+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-index-versioning.1 state=0x562aafd47b10 s->prefetch_data=0 2019-12-11T16:45:37.906 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.922+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:37.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.940+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x16 2019-12-11T16:45:37.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.940+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2019-12-11T16:45:37.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.940+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:37.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.940+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:37.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.940+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:37.925 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.940+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 from cache 2019-12-11T16:45:37.925 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.940+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=193 2019-12-11T16:45:37.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:37.959+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584968 cookie 94741337890976 notifier 4475 bl.length()=193 2019-12-11T16:45:38.097 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.113+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4 from cache 2019-12-11T16:45:38.097 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.113+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=250 2019-12-11T16:45:38.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.121+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584962 cookie 94741337887792 notifier 4475 bl.length()=250 2019-12-11T16:45:38.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.659+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-simple.1 state=0x562aafd490b0 s->prefetch_data=0 2019-12-11T16:45:38.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.659+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : miss 2019-12-11T16:45:38.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.675+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x6 2019-12-11T16:45:38.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.675+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2019-12-11T16:45:38.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.675+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:38.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.675+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:38.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.675+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:38.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.675+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:38.679 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.695+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-11T16:45:38.679 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.695+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x1 2019-12-11T16:45:38.679 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.695+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2019-12-11T16:45:38.679 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.695+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 state=0x562aafd4a750 s->prefetch_data=0 2019-12-11T16:45:38.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.695+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 : miss 2019-12-11T16:45:38.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 info.flags=0x16 2019-12-11T16:45:38.697 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 to cache LRU end 2019-12-11T16:45:38.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:38.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:38.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:38.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:38.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:38.699 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:38.699 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.713+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:38.723 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.739+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-11T16:45:38.723 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.739+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 info.flags=0x13 2019-12-11T16:45:38.723 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.739+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 to cache LRU end 2019-12-11T16:45:38.723 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.739+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:38.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.739+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:38.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.739+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 2019-12-11T16:45:38.740 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.757+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-simple.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5]) start [] num_entries 1100 2019-12-11T16:45:38.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.775+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo[] 2019-12-11T16:45:38.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.775+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-simple.1:foo state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:38.768 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.784+0000 7f7fc3b64ec0 10 manifest: total_size = 26078 2019-12-11T16:45:38.768 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.784+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.71 2019-12-11T16:45:38.769 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.784+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-multipart-simple.1:foo state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:38.769 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.784+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:38.769 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.784+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 2019-12-11T16:45:38.902 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=7673 rules.size()=3 2019-12-11T16:45:38.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=7673 2019-12-11T16:45:38.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=7673 stripe_ofs=7673 part_ofs=7673 rule->part_size=7669 2019-12-11T16:45:38.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=7669 rules.size()=3 2019-12-11T16:45:38.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4197881 part_ofs=7673 rule->part_size=7669 2019-12-11T16:45:38.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=15342 stripe_ofs=15342 part_ofs=15342 rule->part_size=7669 2019-12-11T16:45:38.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=7669 rules.size()=3 2019-12-11T16:45:38.904 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4205550 part_ofs=15342 rule->part_size=7669 2019-12-11T16:45:38.904 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=23011 stripe_ofs=23011 part_ofs=23011 rule->part_size=3067 2019-12-11T16:45:38.904 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=3067 rules.size()=3 2019-12-11T16:45:38.904 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4213219 part_ofs=23011 rule->part_size=3067 2019-12-11T16:45:38.904 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=26078 stripe_ofs=26078 part_ofs=26078 rule->part_size=3067 2019-12-11T16:45:38.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.918+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.71 2019-12-11T16:45:38.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.988+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-simple.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5]) start _multipart_[] num_entries 1001 2019-12-11T16:45:38.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.988+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:38.983 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:38.999+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-simple.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:39.073 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.089+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-simple.1 state=0x562aafbecb70 s->prefetch_data=0 2019-12-11T16:45:39.073 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.089+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:39.090 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.106+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x16 2019-12-11T16:45:39.091 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.107+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2019-12-11T16:45:39.091 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.107+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:39.091 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.107+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:39.091 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.107+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:39.091 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.107+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 from cache 2019-12-11T16:45:39.092 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.107+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=183 2019-12-11T16:45:39.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.125+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584963 cookie 94741337909264 notifier 4475 bl.length()=183 2019-12-11T16:45:39.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.640+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5 from cache 2019-12-11T16:45:39.625 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.640+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=240 2019-12-11T16:45:39.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:39.661+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584961 cookie 94741337893008 notifier 4475 bl.length()=240 2019-12-11T16:45:40.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.096+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-copy.1 state=0x562aafd490b0 s->prefetch_data=0 2019-12-11T16:45:40.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.096+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : miss 2019-12-11T16:45:40.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.106+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x6 2019-12-11T16:45:40.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.106+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2019-12-11T16:45:40.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.106+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:40.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.106+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:40.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.106+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:40.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.106+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:40.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.115+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=227 2019-12-11T16:45:40.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.115+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x1 2019-12-11T16:45:40.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.115+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2019-12-11T16:45:40.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.115+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 state=0x562aafd4a750 s->prefetch_data=0 2019-12-11T16:45:40.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.115+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 : miss 2019-12-11T16:45:40.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 info.flags=0x16 2019-12-11T16:45:40.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 to cache LRU end 2019-12-11T16:45:40.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:40.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:40.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:40.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:40.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:40.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:40.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.125+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.143+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=319 2019-12-11T16:45:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.143+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 info.flags=0x13 2019-12-11T16:45:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.143+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 to cache LRU end 2019-12-11T16:45:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.143+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:40.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.143+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:40.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.143+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:40.137 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.153+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class-copy.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7]) start [] num_entries 1100 2019-12-11T16:45:40.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-11T16:45:40.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-FROZEN[] 2019-12-11T16:45:40.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM[] 2019-12-11T16:45:40.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-STANDARD[] 2019-12-11T16:45:40.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-11T16:45:40.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN[] 2019-12-11T16:45:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM[] 2019-12-11T16:45:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD[] 2019-12-11T16:45:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-11T16:45:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-FROZEN[] 2019-12-11T16:45:40.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM[] 2019-12-11T16:45:40.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-STANDARD[] 2019-12-11T16:45:40.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.170+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.179+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:40.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.179+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.108 2019-12-11T16:45:40.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.179+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.179+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:40.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.179+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:40.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.293+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:40.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.293+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:40.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.293+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:40.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.293+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:40.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.293+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.31tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.108 2019-12-11T16:45:40.341 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.357+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.354 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.368+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.368+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.109 2019-12-11T16:45:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.368+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.368+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.368+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:40.477 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.492+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:40.477 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.492+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:40.477 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.492+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:40.478 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.492+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:40.478 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.493+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.15tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.109 2019-12-11T16:45:40.541 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.556+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.553 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.565+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:40.553 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.565+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _hU_i6aZMruJxPsmUGMxI2-jdLGVRfvA 2019-12-11T16:45:40.553 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.565+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.553 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.565+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:40.554 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.565+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:40.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.702+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:40.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.702+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:40.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.702+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:40.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.702+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:40.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.702+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.14tag is: _hU_i6aZMruJxPsmUGMxI2-jdLGVRfvA 2019-12-11T16:45:40.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.778+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.772 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.788+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:40.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.788+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _ReMcfuI01taIs4f1MGQH-6DTLEP-nKz 2019-12-11T16:45:40.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.788+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.788+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:40.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.788+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:40.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.939+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:40.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.939+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:40.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.939+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.20tag is: _ReMcfuI01taIs4f1MGQH-6DTLEP-nKz 2019-12-11T16:45:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:40.991+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.007+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:40.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.007+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.112 2019-12-11T16:45:40.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.007+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x562aafd498a8 s->prefetch_data=0 2019-12-11T16:45:40.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.007+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd498a8 2019-12-11T16:45:40.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.007+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:41.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.128+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:41.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.128+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:41.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.128+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:41.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.128+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:41.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.128+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.112 2019-12-11T16:45:41.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.205+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.216+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:41.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.216+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _CqUZHVLLrh2zmwk8_4b8hbt4LAxOW_v 2019-12-11T16:45:41.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.216+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.216+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:41.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.216+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:41.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.359+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:41.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.359+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:41.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.359+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:41.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.359+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:41.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.359+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.25tag is: _CqUZHVLLrh2zmwk8_4b8hbt4LAxOW_v 2019-12-11T16:45:41.404 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.420+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.414 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.430+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:41.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.430+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.114 2019-12-11T16:45:41.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.430+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.430+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:41.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.430+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:41.516 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.532+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:41.516 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.532+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:41.516 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.532+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:41.517 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.532+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:41.517 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.532+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.114 2019-12-11T16:45:41.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.620+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.613 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.629+0000 7f7fc3b64ec0 10 manifest: total_size = 5242880 2019-12-11T16:45:41.613 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.629+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _8iZnx_41L4DAYbUNsC3SsWHnqt4ca7H 2019-12-11T16:45:41.613 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.629+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.613 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.629+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:41.614 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.629+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:41.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.751+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:41.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.751+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-11T16:45:41.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.751+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: _8iZnx_41L4DAYbUNsC3SsWHnqt4ca7H 2019-12-11T16:45:41.840 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.857+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.849 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.866+0000 7f7fc3b64ec0 10 manifest: total_size = 22651 2019-12-11T16:45:41.850 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.866+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.116 2019-12-11T16:45:41.850 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.866+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.850 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.866+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:41.850 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.866+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:41.959 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.975+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.970 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.987+0000 7f7fc3b64ec0 10 manifest: total_size = 22651 2019-12-11T16:45:41.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.987+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _8ulQ3CnLOgSdmGeKUAW8-YJ9Am_dmil 2019-12-11T16:45:41.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.987+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:41.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.987+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:41.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:41.987+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.149+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.149+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=22651 stripe_ofs=22651 part_ofs=0 rule->part_size=0 2019-12-11T16:45:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.149+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: _8ulQ3CnLOgSdmGeKUAW8-YJ9Am_dmil 2019-12-11T16:45:42.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.222+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:42.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.232+0000 7f7fc3b64ec0 10 manifest: total_size = 22651 2019-12-11T16:45:42.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.232+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _1-85uefwqjc5-0MvB3JMFhwLM4835Km 2019-12-11T16:45:42.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.232+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:42.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.232+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:42.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.232+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:42.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.346+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:42.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.346+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=22651 stripe_ofs=22651 part_ofs=0 rule->part_size=0 2019-12-11T16:45:42.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.346+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: _1-85uefwqjc5-0MvB3JMFhwLM4835Km 2019-12-11T16:45:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.401+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.409+0000 7f7fc3b64ec0 10 manifest: total_size = 22651 2019-12-11T16:45:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.409+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _nQ75ND-GYIZdgUN3edIQ3KCZx0vDsfb 2019-12-11T16:45:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.409+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.409+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.409+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 2019-12-11T16:45:42.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.525+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class-copy.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7]) start _multipart_[] num_entries 1001 2019-12-11T16:45:42.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.525+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:42.573 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.589+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-copy.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:42.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.678+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-copy.1 state=0x562aafd490b0 s->prefetch_data=0 2019-12-11T16:45:42.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.678+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:42.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.692+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x16 2019-12-11T16:45:42.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.692+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2019-12-11T16:45:42.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.692+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:42.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.692+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:42.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.692+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:42.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.692+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 from cache 2019-12-11T16:45:42.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.692+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=189 2019-12-11T16:45:42.694 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:42.710+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584969 cookie 94741337890976 notifier 4475 bl.length()=189 2019-12-11T16:45:43.113 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.129+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7 from cache 2019-12-11T16:45:43.113 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.129+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=246 2019-12-11T16:45:43.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.141+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584963 cookie 94741337887792 notifier 4475 bl.length()=246 2019-12-11T16:45:43.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.504+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart-copy.1 state=0x562aafd490b0 s->prefetch_data=0 2019-12-11T16:45:43.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.505+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : miss 2019-12-11T16:45:43.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.523+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x6 2019-12-11T16:45:43.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.523+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2019-12-11T16:45:43.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.523+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:43.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.523+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:43.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.523+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:43.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.523+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:43.521 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.537+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=237 2019-12-11T16:45:43.521 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.537+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x1 2019-12-11T16:45:43.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.537+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2019-12-11T16:45:43.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.537+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 state=0x562aafd49fa0 s->prefetch_data=0 2019-12-11T16:45:43.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.537+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 : miss 2019-12-11T16:45:43.548 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 info.flags=0x16 2019-12-11T16:45:43.548 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 to cache LRU end 2019-12-11T16:45:43.548 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:43.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:43.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:43.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:43.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:43.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:43.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.564+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:43.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.588+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=329 2019-12-11T16:45:43.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.588+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 info.flags=0x13 2019-12-11T16:45:43.573 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.588+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 to cache LRU end 2019-12-11T16:45:43.573 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.588+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:43.574 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.588+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:43.574 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.588+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:43.593 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.609+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class-multipart-copy.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9]) start [] num_entries 1100 2019-12-11T16:45:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-11T16:45:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-FROZEN[] 2019-12-11T16:45:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM[] 2019-12-11T16:45:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-STANDARD[] 2019-12-11T16:45:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-11T16:45:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN[] 2019-12-11T16:45:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM[] 2019-12-11T16:45:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD[] 2019-12-11T16:45:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-11T16:45:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-FROZEN[] 2019-12-11T16:45:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM[] 2019-12-11T16:45:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-STANDARD[] 2019-12-11T16:45:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.656+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:43.654 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.670+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:43.654 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.670+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.224 2019-12-11T16:45:43.655 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.670+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:43.655 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.670+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:43.655 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.670+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:43.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:43.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:43.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:43.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:43.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:43.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:43.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:43.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:43.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:43.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:43.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:43.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:43.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-11T16:45:43.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:43.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.837+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-11T16:45:43.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.838+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.224 2019-12-11T16:45:43.888 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.905+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:43.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.917+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:43.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.917+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.225 2019-12-11T16:45:43.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.917+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:43.902 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.917+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:43.902 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:43.917+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:44.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.024 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:44.024 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:44.024 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.024 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:44.024 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:44.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-11T16:45:44.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:44.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-11T16:45:44.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.039+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.225 2019-12-11T16:45:44.073 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.090+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.101+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:44.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.101+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _2kF6UOJs0-s2jUp51V2Sdhl8ekQMlOP 2019-12-11T16:45:44.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.101+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.101+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:44.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.101+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:44.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.224+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.224+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.224+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.224+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.224+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.224+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.224+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: _2kF6UOJs0-s2jUp51V2Sdhl8ekQMlOP 2019-12-11T16:45:44.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.279+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.288 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.293+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:44.288 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.293+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _elkAnGy4vt6OWAcRHgAzyhcKRCe06Ij 2019-12-11T16:45:44.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.293+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.293+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:44.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.293+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:44.399 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.416+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.416+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.416+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.416+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.416+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: _elkAnGy4vt6OWAcRHgAzyhcKRCe06Ij 2019-12-11T16:45:44.461 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.477+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.468 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.485+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:44.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.485+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.233 2019-12-11T16:45:44.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.485+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.485+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:44.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.485+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:44.575 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.590+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.575 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.590+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:44.575 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.590+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:44.575 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.590+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.590+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:44.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.590+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.590+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:44.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:44.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:44.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-11T16:45:44.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:44.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-11T16:45:44.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.591+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.233 2019-12-11T16:45:44.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.658+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.652 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.668+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:44.652 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.668+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _epVPX-oNh_mArfKkvH03IkCWvT5ehfS 2019-12-11T16:45:44.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.668+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:44.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.668+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:44.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.668+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:44.844 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.860+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.844 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.860+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.860+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.860+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.860+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:44.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.860+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-11T16:45:44.846 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:44.860+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: _epVPX-oNh_mArfKkvH03IkCWvT5ehfS 2019-12-11T16:45:44.985 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.001+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.030+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.030+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.235 2019-12-11T16:45:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.030+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.030+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.030+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:45.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:45.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:45.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:45.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:45.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:45.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:45.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:45.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:45.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:45.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:45.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:45.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:45.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-11T16:45:45.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:45.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-11T16:45:45.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.145+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.235 2019-12-11T16:45:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.264+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.278+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.278+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _u5uRLbrOQbKGo6mSzwHIHtfYuwaOwUi 2019-12-11T16:45:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.278+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.278+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.278+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:45.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.409+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:45.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.409+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-11T16:45:45.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.409+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:45.393 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.409+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-11T16:45:45.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.409+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: _u5uRLbrOQbKGo6mSzwHIHtfYuwaOwUi 2019-12-11T16:45:45.465 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.482+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.492+0000 7f7fc3b64ec0 10 manifest: total_size = 541681 2019-12-11T16:45:45.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.492+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.242 2019-12-11T16:45:45.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.492+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.492+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:45.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.492+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:45.567 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2019-12-11T16:45:45.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2019-12-11T16:45:45.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2019-12-11T16:45:45.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2019-12-11T16:45:45.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4436455 part_ofs=246247 rule->part_size=246247 2019-12-11T16:45:45.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=49187 2019-12-11T16:45:45.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=49187 rules.size()=2 2019-12-11T16:45:45.569 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4682702 part_ofs=492494 rule->part_size=49187 2019-12-11T16:45:45.569 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=541681 rule->part_size=49187 2019-12-11T16:45:45.569 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.584+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.12tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.242 2019-12-11T16:45:45.637 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.654+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.662+0000 7f7fc3b64ec0 10 manifest: total_size = 541681 2019-12-11T16:45:45.646 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.662+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _nwwDKOCnpgi6PvawNQGGb6Qav5S7tEs 2019-12-11T16:45:45.646 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.662+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.646 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.662+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:45.646 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.662+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:45.748 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.764+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:45.748 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.764+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=0 rule->part_size=0 2019-12-11T16:45:45.748 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.764+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: _nwwDKOCnpgi6PvawNQGGb6Qav5S7tEs 2019-12-11T16:45:45.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.825+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.821 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.838+0000 7f7fc3b64ec0 10 manifest: total_size = 541681 2019-12-11T16:45:45.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.838+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to _3tBuxl1qoU23OGFJu0bsB-R5gAI19A0 2019-12-11T16:45:45.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.838+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:45.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.838+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:45.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.838+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:45.928 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.945+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:45.929 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.945+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=0 rule->part_size=0 2019-12-11T16:45:45.929 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:45.945+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: _3tBuxl1qoU23OGFJu0bsB-R5gAI19A0 2019-12-11T16:45:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.014+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:46.010 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.027+0000 7f7fc3b64ec0 10 manifest: total_size = 541681 2019-12-11T16:45:46.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.027+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.245 2019-12-11T16:45:46.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.027+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:46.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.027+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:46.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.027+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 2019-12-11T16:45:46.117 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2019-12-11T16:45:46.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2019-12-11T16:45:46.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2019-12-11T16:45:46.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2019-12-11T16:45:46.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4436455 part_ofs=246247 rule->part_size=246247 2019-12-11T16:45:46.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=49187 2019-12-11T16:45:46.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=49187 rules.size()=2 2019-12-11T16:45:46.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4682702 part_ofs=492494 rule->part_size=49187 2019-12-11T16:45:46.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=541681 rule->part_size=49187 2019-12-11T16:45:46.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.134+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.245 2019-12-11T16:45:46.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.217+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class-multipart-copy.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9]) start _multipart_[] num_entries 1001 2019-12-11T16:45:46.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.217+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:46.234 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.251+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart-copy.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:46.301 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.317+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart-copy.1 state=0x562aafd4a460 s->prefetch_data=0 2019-12-11T16:45:46.302 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.317+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:46.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.328+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x16 2019-12-11T16:45:46.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.328+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2019-12-11T16:45:46.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.328+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:46.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.328+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:46.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.328+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:46.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.328+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 from cache 2019-12-11T16:45:46.313 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.328+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=199 2019-12-11T16:45:46.327 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.344+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584962 cookie 94741337893008 notifier 4475 bl.length()=199 2019-12-11T16:45:46.406 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.423+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9 from cache 2019-12-11T16:45:46.407 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.423+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=256 2019-12-11T16:45:46.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.432+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584970 cookie 94741337890976 notifier 4475 bl.length()=256 2019-12-11T16:45:46.871 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.888+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart.1 state=0x562aafd4a3e0 s->prefetch_data=0 2019-12-11T16:45:46.872 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.888+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : miss 2019-12-11T16:45:46.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.898+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x6 2019-12-11T16:45:46.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.898+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2019-12-11T16:45:46.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.898+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:46.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.898+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:46.883 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.898+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:46.883 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.898+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:46.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.911+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=232 2019-12-11T16:45:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.911+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x1 2019-12-11T16:45:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.911+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2019-12-11T16:45:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.911+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 state=0x562aafbece50 s->prefetch_data=0 2019-12-11T16:45:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.911+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 : miss 2019-12-11T16:45:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 info.flags=0x16 2019-12-11T16:45:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 to cache LRU end 2019-12-11T16:45:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.921+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.940+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=324 2019-12-11T16:45:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.940+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 info.flags=0x13 2019-12-11T16:45:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.940+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 to cache LRU end 2019-12-11T16:45:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.940+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.940+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.940+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 2019-12-11T16:45:46.934 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.951+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class-multipart.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8]) start [] num_entries 1100 2019-12-11T16:45:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.979+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-11T16:45:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.979+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-11T16:45:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.979+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-11T16:45:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.979+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:46.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.987+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:46.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.987+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.196 2019-12-11T16:45:46.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.987+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:46.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.987+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:46.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:46.987+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 2019-12-11T16:45:47.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:47.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:47.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:47.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:47.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-11T16:45:47.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:47.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.082+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-11T16:45:47.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.083+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.196 2019-12-11T16:45:47.149 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.165+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:47.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.173+0000 7f7fc3b64ec0 10 manifest: total_size = 11534336 2019-12-11T16:45:47.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.173+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.203 2019-12-11T16:45:47.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.173+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:47.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.174+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.174+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 2019-12-11T16:45:47.274 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:47.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:47.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-11T16:45:47.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-11T16:45:47.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-11T16:45:47.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:47.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-11T16:45:47.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-11T16:45:47.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-11T16:45:47.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.290+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.203 2019-12-11T16:45:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.336+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:47.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.346+0000 7f7fc3b64ec0 10 manifest: total_size = 45360 2019-12-11T16:45:47.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.346+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.210 2019-12-11T16:45:47.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.346+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x562aafd4cdf8 s->prefetch_data=0 2019-12-11T16:45:47.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.346+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafd4cdf8 2019-12-11T16:45:47.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.346+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 2019-12-11T16:45:47.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2019-12-11T16:45:47.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=20613 2019-12-11T16:45:47.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2019-12-11T16:45:47.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2019-12-11T16:45:47.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4210821 part_ofs=20613 rule->part_size=20613 2019-12-11T16:45:47.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=4134 2019-12-11T16:45:47.410 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=4134 rules.size()=2 2019-12-11T16:45:47.410 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): stripe_ofs=4231434 part_ofs=41226 rule->part_size=4134 2019-12-11T16:45:47.410 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=45360 stripe_ofs=45360 part_ofs=45360 rule->part_size=4134 2019-12-11T16:45:47.410 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.425+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.9tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.210 2019-12-11T16:45:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.465+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class-multipart.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8]) start _multipart_[] num_entries 1001 2019-12-11T16:45:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.465+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:47.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.488+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:47.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.548+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart.1 state=0x562aafd4a3e0 s->prefetch_data=0 2019-12-11T16:45:47.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.548+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:47.542 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.558+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x16 2019-12-11T16:45:47.542 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.558+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2019-12-11T16:45:47.542 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.558+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:47.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.558+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:47.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.558+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:47.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.558+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 from cache 2019-12-11T16:45:47.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.558+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=194 2019-12-11T16:45:47.554 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.570+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584971 cookie 94741337911440 notifier 4475 bl.length()=194 2019-12-11T16:45:47.933 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.949+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8 from cache 2019-12-11T16:45:47.933 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.949+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=251 2019-12-11T16:45:47.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:47.964+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584972 cookie 94741337890976 notifier 4475 bl.length()=251 2019-12-11T16:45:48.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.368+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class.1 state=0x562aafbece50 s->prefetch_data=0 2019-12-11T16:45:48.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.368+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : miss 2019-12-11T16:45:48.361 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.377+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x6 2019-12-11T16:45:48.361 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.377+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2019-12-11T16:45:48.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.377+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:48.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.377+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:48.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.377+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:48.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.377+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:48.371 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.388+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=222 2019-12-11T16:45:48.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.388+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x1 2019-12-11T16:45:48.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.388+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2019-12-11T16:45:48.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.388+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 state=0x562aafd4a3e0 s->prefetch_data=0 2019-12-11T16:45:48.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.388+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 : miss 2019-12-11T16:45:48.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 info.flags=0x16 2019-12-11T16:45:48.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 to cache LRU end 2019-12-11T16:45:48.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:48.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:48.384 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:48.384 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:48.384 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:48.384 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:48.384 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.399+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.415+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=314 2019-12-11T16:45:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.415+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 info.flags=0x13 2019-12-11T16:45:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.415+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 to cache LRU end 2019-12-11T16:45:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.415+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.415+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.415+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 2019-12-11T16:45:48.414 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.430+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6]) start [] num_entries 1100 2019-12-11T16:45:48.431 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.448+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-11T16:45:48.431 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.448+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-11T16:45:48.432 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.448+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-11T16:45:48.432 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.448+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:48.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.464+0000 7f7fc3b64ec0 10 manifest: total_size = 10485760 2019-12-11T16:45:48.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.464+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.84 2019-12-11T16:45:48.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.464+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:48.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.464+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafcae078 2019-12-11T16:45:48.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.464+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 2019-12-11T16:45:48.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.625+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:48.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.625+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:48.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.625+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:48.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.625+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-11T16:45:48.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.625+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:48.610 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.625+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2019-12-11T16:45:48.610 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.625+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.7tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.84 2019-12-11T16:45:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.694+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:48.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.706+0000 7f7fc3b64ec0 10 manifest: total_size = 10485760 2019-12-11T16:45:48.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.706+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.85 2019-12-11T16:45:48.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.706+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:48.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.706+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafcae078 2019-12-11T16:45:48.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.706+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 2019-12-11T16:45:48.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.800+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:48.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.800+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-11T16:45:48.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.800+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:48.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.800+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-11T16:45:48.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.800+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-11T16:45:48.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.800+0000 7f7fc3b64ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2019-12-11T16:45:48.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.800+0000 7f7fc3b64ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.85 2019-12-11T16:45:48.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.858+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:48.851 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.868+0000 7f7fc3b64ec0 10 manifest: total_size = 44258 2019-12-11T16:45:48.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.868+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.86 2019-12-11T16:45:48.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.868+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:48.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.868+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafcae078 2019-12-11T16:45:48.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.868+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 2019-12-11T16:45:48.956 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.972+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6]) start _multipart_[] num_entries 1001 2019-12-11T16:45:48.956 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:48.972+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:49.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.020+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:49.062 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.079+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class.1 state=0x562aafd4a3e0 s->prefetch_data=0 2019-12-11T16:45:49.062 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.079+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.091+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x16 2019-12-11T16:45:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.091+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2019-12-11T16:45:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.091+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.091+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.091+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.091+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 from cache 2019-12-11T16:45:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.091+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=184 2019-12-11T16:45:49.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.101+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584973 cookie 94741337890976 notifier 4475 bl.length()=184 2019-12-11T16:45:49.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.263+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6 from cache 2019-12-11T16:45:49.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.263+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=241 2019-12-11T16:45:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.281+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584961 cookie 94741337929168 notifier 4475 bl.length()=241 2019-12-11T16:45:49.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.682+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.1 state=0x562aafd4a3e0 s->prefetch_data=0 2019-12-11T16:45:49.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.682+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : miss 2019-12-11T16:45:49.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.717+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x6 2019-12-11T16:45:49.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.717+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2019-12-11T16:45:49.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.717+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:49.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.717+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:49.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.718+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:49.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.718+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:49.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.740+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-11T16:45:49.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.740+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x1 2019-12-11T16:45:49.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.740+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2019-12-11T16:45:49.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.740+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 state=0x562aafbece50 s->prefetch_data=0 2019-12-11T16:45:49.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.740+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 : miss 2019-12-11T16:45:49.740 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 info.flags=0x16 2019-12-11T16:45:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 to cache LRU end 2019-12-11T16:45:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:49.742 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:49.742 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.756+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:49.756 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.772+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-11T16:45:49.756 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.772+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 info.flags=0x13 2019-12-11T16:45:49.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.772+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 to cache LRU end 2019-12-11T16:45:49.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.772+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:49.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.772+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:49.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.772+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 2019-12-11T16:45:49.771 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.787+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-small-obj-data.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10]) start [] num_entries 1100 2019-12-11T16:45:49.802 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.818+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got __[] 2019-12-11T16:45:49.802 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.818+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got __ _[] 2019-12-11T16:45:49.802 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.818+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got ___[] 2019-12-11T16:45:49.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.818+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got obj[] 2019-12-11T16:45:49.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.818+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:__ state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:49.814 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.831+0000 7f7fc3b64ec0 10 manifest: total_size = 49187 2019-12-11T16:45:49.814 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.831+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.305 2019-12-11T16:45:49.815 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.831+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:__ state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:49.815 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.831+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafcae078 2019-12-11T16:45:49.815 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.831+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 2019-12-11T16:45:49.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.932+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.943+0000 7f7fc3b64ec0 10 manifest: total_size = 49187 2019-12-11T16:45:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.943+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.307 2019-12-11T16:45:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.943+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.943+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafcae078 2019-12-11T16:45:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:49.943+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 2019-12-11T16:45:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.106+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:___ state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.117+0000 7f7fc3b64ec0 10 manifest: total_size = 1048576 2019-12-11T16:45:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.117+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.306 2019-12-11T16:45:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.117+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:___ state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.117+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafcae078 2019-12-11T16:45:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.117+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 2019-12-11T16:45:50.230 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.246+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:obj state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:50.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.259+0000 7f7fc3b64ec0 10 manifest: total_size = 1035 2019-12-11T16:45:50.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.259+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.304 2019-12-11T16:45:50.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.259+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.1:obj state=0x562aafcae078 s->prefetch_data=0 2019-12-11T16:45:50.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.259+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafcae078 2019-12-11T16:45:50.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.259+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 2019-12-11T16:45:50.376 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.393+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-small-obj-data.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10]) start _multipart_[] num_entries 1001 2019-12-11T16:45:50.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.393+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:50.434 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.450+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.1[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:50.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.506+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.1 state=0x562aafbece50 s->prefetch_data=0 2019-12-11T16:45:50.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.506+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:50.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.516+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x16 2019-12-11T16:45:50.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.516+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2019-12-11T16:45:50.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.516+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:50.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.516+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:50.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.516+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:50.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.516+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 from cache 2019-12-11T16:45:50.501 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.516+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=181 2019-12-11T16:45:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:50.525+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584964 cookie 94741337887792 notifier 4475 bl.length()=181 2019-12-11T16:45:51.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.135+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.10 from cache 2019-12-11T16:45:51.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.135+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=239 2019-12-11T16:45:51.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.143+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584965 cookie 94741337887792 notifier 4475 bl.length()=239 2019-12-11T16:45:51.708 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.725+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.2 state=0x562aafbece50 s->prefetch_data=0 2019-12-11T16:45:51.709 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.725+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : miss 2019-12-11T16:45:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.743+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x6 2019-12-11T16:45:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.743+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2019-12-11T16:45:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.743+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.743+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.743+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.743+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.759+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-11T16:45:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.759+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x1 2019-12-11T16:45:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.759+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2019-12-11T16:45:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.759+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 state=0x562aafd4a3e0 s->prefetch_data=0 2019-12-11T16:45:51.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.759+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 : miss 2019-12-11T16:45:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 info.flags=0x16 2019-12-11T16:45:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 to cache LRU end 2019-12-11T16:45:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.775+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:51.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.800+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-11T16:45:51.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.800+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 info.flags=0x13 2019-12-11T16:45:51.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.800+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 to cache LRU end 2019-12-11T16:45:51.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.800+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:51.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.800+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:51.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.800+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 2019-12-11T16:45:51.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.825+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-small-obj-data.2[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11]) start [] num_entries 1100 2019-12-11T16:45:51.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.859+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got __[] 2019-12-11T16:45:51.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.859+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got __ _[] 2019-12-11T16:45:51.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.859+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got ___[] 2019-12-11T16:45:51.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.859+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got obj[] 2019-12-11T16:45:51.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.859+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:__ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:51.854 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.870+0000 7f7fc3b64ec0 10 manifest: total_size = 2078 2019-12-11T16:45:51.854 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.870+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.312 2019-12-11T16:45:51.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.870+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:__ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:51.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.870+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:51.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:51.871+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 2019-12-11T16:45:52.037 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.054+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:52.057 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.074+0000 7f7fc3b64ec0 10 manifest: total_size = 527 2019-12-11T16:45:52.058 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.074+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.314 2019-12-11T16:45:52.058 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.074+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:52.058 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.074+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:52.058 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.074+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 2019-12-11T16:45:52.138 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.154+0000 7f7f817e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:45:52.239 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.255+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:___ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:52.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.264+0000 7f7fc3b64ec0 10 manifest: total_size = 2078 2019-12-11T16:45:52.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.264+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.313 2019-12-11T16:45:52.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.264+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:___ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:52.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.264+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:52.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.264+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 2019-12-11T16:45:52.407 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.423+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:obj state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:52.417 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.433+0000 7f7fc3b64ec0 10 manifest: total_size = 24595 2019-12-11T16:45:52.417 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.433+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.311 2019-12-11T16:45:52.417 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.433+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.2:obj state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:52.418 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.433+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:52.418 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.433+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 2019-12-11T16:45:52.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.616+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-small-obj-data.2[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11]) start _multipart_[] num_entries 1001 2019-12-11T16:45:52.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.616+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:52.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.679+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.2[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:52.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.741+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.2 state=0x562aafbec400 s->prefetch_data=0 2019-12-11T16:45:52.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.741+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:52.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.760+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x16 2019-12-11T16:45:52.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.760+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2019-12-11T16:45:52.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.760+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:52.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.760+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:52.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.760+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:52.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.760+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 from cache 2019-12-11T16:45:52.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.760+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=181 2019-12-11T16:45:52.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:52.774+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584966 cookie 94741337887792 notifier 4475 bl.length()=181 2019-12-11T16:45:53.271 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.288+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.11 from cache 2019-12-11T16:45:53.272 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.288+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=239 2019-12-11T16:45:53.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.298+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584967 cookie 94741337887792 notifier 4475 bl.length()=239 2019-12-11T16:45:53.626 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.642+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.3 state=0x562aafbece50 s->prefetch_data=0 2019-12-11T16:45:53.626 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.642+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : miss 2019-12-11T16:45:53.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.658+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x6 2019-12-11T16:45:53.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.658+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2019-12-11T16:45:53.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.658+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:53.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.658+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:53.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.658+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : type miss (requested=0x1, cached=0x6) 2019-12-11T16:45:53.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.658+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.679+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-11T16:45:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.679+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x1 2019-12-11T16:45:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.679+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2019-12-11T16:45:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.679+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42dea5f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 state=0x562aafd4a3e0 s->prefetch_data=0 2019-12-11T16:45:53.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.679+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 : miss 2019-12-11T16:45:53.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 info.flags=0x16 2019-12-11T16:45:53.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 to cache LRU end 2019-12-11T16:45:53.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:53.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:53.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:53.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.acl 2019-12-11T16:45:53.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 20 Read xattr: user.rgw.idtag 2019-12-11T16:45:53.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 : type miss (requested=0x13, cached=0x16) 2019-12-11T16:45:53.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.699+0000 7f7fc3b64ec0 20 rados->read ofs=0 len=0 2019-12-11T16:45:53.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.723+0000 7f7fc3b64ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-11T16:45:53.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.723+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 info.flags=0x13 2019-12-11T16:45:53.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.723+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 to cache LRU end 2019-12-11T16:45:53.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.723+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:53.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.723+0000 7f7fc3b64ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-11T16:45:53.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.723+0000 7f7fc3b64ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 2019-12-11T16:45:53.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.741+0000 7f7fc3b64ec0 10 cls_bucket_list_unordered :test-client.0-r-test-small-obj-data.3[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12]) start [] num_entries 1100 2019-12-11T16:45:53.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.770+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got __[] 2019-12-11T16:45:53.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.770+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got __ _[] 2019-12-11T16:45:53.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.770+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got ___[] 2019-12-11T16:45:53.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.770+0000 7f7fc3b64ec0 10 RGWRados::cls_bucket_list_unordered: got obj[] 2019-12-11T16:45:53.755 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.770+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:__ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:53.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.779+0000 7f7fc3b64ec0 10 manifest: total_size = 0 2019-12-11T16:45:53.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.779+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.319 2019-12-11T16:45:53.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.779+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:__ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:53.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.779+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:53.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.779+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 2019-12-11T16:45:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.885+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.895+0000 7f7fc3b64ec0 10 manifest: total_size = 0 2019-12-11T16:45:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.895+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.321 2019-12-11T16:45:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.895+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.895+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:53.895+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 2019-12-11T16:45:54.001 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.018+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:___ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:54.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.039+0000 7f7fc3b64ec0 10 manifest: total_size = 0 2019-12-11T16:45:54.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.039+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.320 2019-12-11T16:45:54.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.039+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:___ state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:54.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.039+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:54.024 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.039+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 2019-12-11T16:45:54.138 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.155+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:obj state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:54.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.169+0000 7f7fc3b64ec0 10 manifest: total_size = 0 2019-12-11T16:45:54.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.169+0000 7f7fc3b64ec0 20 get_obj_state: setting s->obj_tag to 14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.318 2019-12-11T16:45:54.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.169+0000 7f7fc3b64ec0 20 get_obj_state: rctx=0x7fff42dea610 obj=test-client.0-r-test-small-obj-data.3:obj state=0x562aafbeb1d8 s->prefetch_data=0 2019-12-11T16:45:54.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.169+0000 7f7fc3b64ec0 20 prepare_atomic_modification: state is not atomic. state=0x562aafbeb1d8 2019-12-11T16:45:54.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.169+0000 7f7fc3b64ec0 20 bucket index object: default.rgw.buckets.index:.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 2019-12-11T16:45:54.324 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.340+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered :test-client.0-r-test-small-obj-data.3[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12]) start _multipart_[] num_entries 1001 2019-12-11T16:45:54.324 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.340+0000 7f7fc3b64ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-11T16:45:54.407 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.424+0000 7f7fc3b64ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.3[14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12])"; objs.size()=0; is_truncated=0 2019-12-11T16:45:54.456 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.473+0000 7f7fc3b64ec0 20 get_system_obj_state: rctx=0x7fff42de9fa8 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.3 state=0x562aafcae9b0 s->prefetch_data=0 2019-12-11T16:45:54.457 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.473+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : type miss (requested=0x16, cached=0x3) 2019-12-11T16:45:54.471 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.487+0000 7f7fc3b64ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x16 2019-12-11T16:45:54.471 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.487+0000 7f7fc3b64ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2019-12-11T16:45:54.471 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.487+0000 7f7fc3b64ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T16:45:54.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.487+0000 7f7fc3b64ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T16:45:54.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.487+0000 7f7fc3b64ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : hit (requested=0x11, cached=0x17) 2019-12-11T16:45:54.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.487+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 from cache 2019-12-11T16:45:54.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.487+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=181 2019-12-11T16:45:54.480 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:54.496+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584968 cookie 94741337887792 notifier 4475 bl.length()=181 2019-12-11T16:45:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:55.148+0000 7f7fc3b64ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.12 from cache 2019-12-11T16:45:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:55.148+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=239 2019-12-11T16:45:55.138 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:55.154+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584969 cookie 94741337887792 notifier 4475 bl.length()=239 2019-12-11T16:45:55.668 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:55.684+0000 7f7fc3b64ec0 10 removing key index: TFEPWBRGOFGYOZCNMHZC 2019-12-11T16:45:55.668 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:55.684+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=170 2019-12-11T16:45:55.681 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:55.697+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584974 cookie 94741337890976 notifier 4475 bl.length()=170 2019-12-11T16:45:56.148 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.165+0000 7f7fc3b64ec0 10 removing email index: ragweed.client.0+test@test.test 2019-12-11T16:45:56.148 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.165+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=182 2019-12-11T16:45:56.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.174+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584975 cookie 94741337890976 notifier 4475 bl.length()=182 2019-12-11T16:45:56.437 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.453+0000 7f7fc3b64ec0 10 removing user buckets index 2019-12-11T16:45:56.437 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.453+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=173 2019-12-11T16:45:56.446 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.462+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584976 cookie 94741337911440 notifier 4475 bl.length()=173 2019-12-11T16:45:56.926 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.942+0000 7f7fc3b64ec0 10 removing user index: ragweed.client.0 2019-12-11T16:45:56.926 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.942+0000 7f7fc3b64ec0 10 removing default.rgw.meta+users.uid+ragweed.client.0 from cache 2019-12-11T16:45:56.926 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.942+0000 7f7fc3b64ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=165 2019-12-11T16:45:56.941 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:56.957+0000 7f7f967fc700 10 RGWWatcher::handle_notify() notify_id 579820584964 cookie 94741337909264 notifier 4475 bl.length()=165 2019-12-11T16:45:57.877 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:57.893+0000 7f7fc3b64ec0 20 remove_watcher() i=0 2019-12-11T16:45:57.877 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:57.893+0000 7f7fc3b64ec0 2 removed watcher, disabling cache 2019-12-11T16:45:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:57.946+0000 7f7fc3b64ec0 20 remove_watcher() i=1 2019-12-11T16:45:57.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:57.996+0000 7f7fc3b64ec0 20 remove_watcher() i=2 2019-12-11T16:45:58.043 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:58.059+0000 7f7fc3b64ec0 20 remove_watcher() i=3 2019-12-11T16:45:58.087 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:58.104+0000 7f7fc3b64ec0 20 remove_watcher() i=4 2019-12-11T16:45:58.135 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:58.152+0000 7f7fc3b64ec0 20 remove_watcher() i=5 2019-12-11T16:45:58.215 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:58.232+0000 7f7fc3b64ec0 20 remove_watcher() i=6 2019-12-11T16:45:58.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:45:58.286+0000 7f7fc3b64ec0 20 remove_watcher() i=7 2019-12-11T16:45:58.279 INFO:tasks.ragweed:Removing ragweed... 2019-12-11T16:45:58.279 INFO:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/ragweed 2019-12-11T16:45:58.368 INFO:teuthology.run_tasks:Running task s3tests-java... 2019-12-11T16:45:58.380 DEBUG:tasks.s3tests_java:S3 Tests Java: __INIT__ 2019-12-11T16:45:58.380 DEBUG:tasks.s3tests_java:S3 Tests Java: SETUP 2019-12-11T16:45:58.380 INFO:tasks.s3tests_java:S3 Tests Java: Downloading test suite... 2019-12-11T16:45:58.380 INFO:teuthology.orchestra.run.smithi079:> git clone -b master https://github.com/ceph/java_s3tests.git /home/ubuntu/cephtest/s3-tests-java 2019-12-11T16:45:58.430 INFO:teuthology.orchestra.run.smithi079.stdout:Cloning into '/home/ubuntu/cephtest/s3-tests-java'... 2019-12-11T16:46:02.328 INFO:tasks.s3tests_java:S3 Tests Java: Installing required packages... 2019-12-11T16:46:02.328 INFO:teuthology.orchestra.run.smithi079:> /home/ubuntu/cephtest/s3-tests-java/bootstrap.sh 2019-12-11T16:46:04.888 INFO:teuthology.orchestra.run.smithi079.stdout: Gradle will be installed under /opt/gradle 2019-12-11T16:46:04.936 INFO:teuthology.orchestra.run.smithi079.stderr:--2019-12-11 16:46:04-- https://services.gradle.org/distributions/gradle-4.7-bin.zip 2019-12-11T16:46:04.938 INFO:teuthology.orchestra.run.smithi079.stderr:Resolving services.gradle.org (services.gradle.org)... 104.18.191.9, 104.18.190.9, 2606:4700::6812:bf09, ... 2019-12-11T16:46:04.947 INFO:teuthology.orchestra.run.smithi079.stderr:Connecting to services.gradle.org (services.gradle.org)|104.18.191.9|:443... connected. 2019-12-11T16:46:04.993 INFO:teuthology.orchestra.run.smithi079.stderr:HTTP request sent, awaiting response... 301 Moved Permanently 2019-12-11T16:46:04.993 INFO:teuthology.orchestra.run.smithi079.stderr:Location: https://downloads.gradle-dn.com/distributions/gradle-4.7-bin.zip [following] 2019-12-11T16:46:04.993 INFO:teuthology.orchestra.run.smithi079.stderr:--2019-12-11 16:46:05-- https://downloads.gradle-dn.com/distributions/gradle-4.7-bin.zip 2019-12-11T16:46:04.994 INFO:teuthology.orchestra.run.smithi079.stderr:Resolving downloads.gradle-dn.com (downloads.gradle-dn.com)... 104.17.160.20, 104.17.159.20, 2606:4700::6811:a014, ... 2019-12-11T16:46:05.001 INFO:teuthology.orchestra.run.smithi079.stderr:Connecting to downloads.gradle-dn.com (downloads.gradle-dn.com)|104.17.160.20|:443... connected. 2019-12-11T16:46:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:HTTP request sent, awaiting response... 200 OK 2019-12-11T16:46:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:Length: 75323957 (72M) [application/zip] 2019-12-11T16:46:05.054 INFO:teuthology.orchestra.run.smithi079.stderr:Saving to: ‘gradle-4.7-bin.zip’ 2019-12-11T16:46:05.054 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T16:46:05.063 INFO:teuthology.orchestra.run.smithi079.stderr: 0K .......... .......... .......... .......... .......... 0% 6.50M 11s 2019-12-11T16:46:05.065 INFO:teuthology.orchestra.run.smithi079.stderr: 50K .......... .......... .......... .......... .......... 0% 13.7M 8s 2019-12-11T16:46:05.067 INFO:teuthology.orchestra.run.smithi079.stderr: 100K .......... .......... .......... .......... .......... 0% 10.4M 8s 2019-12-11T16:46:05.070 INFO:teuthology.orchestra.run.smithi079.stderr: 150K .......... .......... .......... .......... .......... 0% 19.1M 7s 2019-12-11T16:46:05.072 INFO:teuthology.orchestra.run.smithi079.stderr: 200K .......... .......... .......... .......... .......... 0% 20.8M 6s 2019-12-11T16:46:05.074 INFO:teuthology.orchestra.run.smithi079.stderr: 250K .......... .......... .......... .......... .......... 0% 24.8M 6s 2019-12-11T16:46:05.076 INFO:teuthology.orchestra.run.smithi079.stderr: 300K .......... .......... .......... .......... .......... 0% 47.0M 5s 2019-12-11T16:46:05.077 INFO:teuthology.orchestra.run.smithi079.stderr: 350K .......... .......... .......... .......... .......... 0% 25.7M 5s 2019-12-11T16:46:05.078 INFO:teuthology.orchestra.run.smithi079.stderr: 400K .......... .......... .......... .......... .......... 0% 33.6M 4s 2019-12-11T16:46:05.080 INFO:teuthology.orchestra.run.smithi079.stderr: 450K .......... .......... .......... .......... .......... 0% 61.9M 4s 2019-12-11T16:46:05.086 INFO:teuthology.orchestra.run.smithi079.stderr: 500K .......... .......... .......... .......... .......... 0% 40.7M 4s 2019-12-11T16:46:05.086 INFO:teuthology.orchestra.run.smithi079.stderr: 550K .......... .......... .......... .......... .......... 0% 48.1M 4s 2019-12-11T16:46:05.086 INFO:teuthology.orchestra.run.smithi079.stderr: 600K .......... .......... .......... .......... .......... 0% 87.5M 3s 2019-12-11T16:46:05.086 INFO:teuthology.orchestra.run.smithi079.stderr: 650K .......... .......... .......... .......... .......... 0% 63.3M 3s 2019-12-11T16:46:05.086 INFO:teuthology.orchestra.run.smithi079.stderr: 700K .......... .......... .......... .......... .......... 1% 61.3M 3s 2019-12-11T16:46:05.087 INFO:teuthology.orchestra.run.smithi079.stderr: 750K .......... .......... .......... .......... .......... 1% 71.2M 3s 2019-12-11T16:46:05.087 INFO:teuthology.orchestra.run.smithi079.stderr: 800K .......... .......... .......... .......... .......... 1% 74.4M 3s 2019-12-11T16:46:05.093 INFO:teuthology.orchestra.run.smithi079.stderr: 850K .......... .......... .......... .......... .......... 1% 19.2M 3s 2019-12-11T16:46:05.093 INFO:teuthology.orchestra.run.smithi079.stderr: 900K .......... .......... .......... .......... .......... 1% 216M 3s 2019-12-11T16:46:05.093 INFO:teuthology.orchestra.run.smithi079.stderr: 950K .......... .......... .......... .......... .......... 1% 220M 3s 2019-12-11T16:46:05.093 INFO:teuthology.orchestra.run.smithi079.stderr: 1000K .......... .......... .......... .......... .......... 1% 320M 3s 2019-12-11T16:46:05.093 INFO:teuthology.orchestra.run.smithi079.stderr: 1050K .......... .......... .......... .......... .......... 1% 349M 2s 2019-12-11T16:46:05.093 INFO:teuthology.orchestra.run.smithi079.stderr: 1100K .......... .......... .......... .......... .......... 1% 288M 2s 2019-12-11T16:46:05.094 INFO:teuthology.orchestra.run.smithi079.stderr: 1150K .......... .......... .......... .......... .......... 1% 119M 2s 2019-12-11T16:46:06.366 INFO:teuthology.orchestra.run.smithi079.stderr: 1200K .......... .......... .......... .......... .......... 1% 90.0M 2s 2019-12-11T16:46:06.366 INFO:teuthology.orchestra.run.smithi079.stderr: 1250K .......... .......... .......... .......... .......... 1% 121M 2s 2019-12-11T16:46:06.366 INFO:teuthology.orchestra.run.smithi079.stderr: 1300K .......... .......... .......... .......... .......... 1% 127M 2s 2019-12-11T16:46:06.367 INFO:teuthology.orchestra.run.smithi079.stderr: 1350K .......... .......... .......... .......... .......... 1% 124M 2s 2019-12-11T16:46:06.367 INFO:teuthology.orchestra.run.smithi079.stderr: 1400K .......... .......... .......... .......... .......... 1% 119M 2s 2019-12-11T16:46:06.367 INFO:teuthology.orchestra.run.smithi079.stderr: 1450K .......... .......... .......... .......... .......... 2% 140M 2s 2019-12-11T16:46:06.367 INFO:teuthology.orchestra.run.smithi079.stderr: 1500K .......... .......... .......... .......... .......... 2% 212M 2s 2019-12-11T16:46:06.368 INFO:teuthology.orchestra.run.smithi079.stderr: 1550K .......... .......... .......... .......... .......... 2% 84.7M 2s 2019-12-11T16:46:06.368 INFO:teuthology.orchestra.run.smithi079.stderr: 1600K .......... .......... .......... .......... .......... 2% 293M 2s 2019-12-11T16:46:06.368 INFO:teuthology.orchestra.run.smithi079.stderr: 1650K .......... .......... .......... .......... .......... 2% 192M 2s 2019-12-11T16:46:06.368 INFO:teuthology.orchestra.run.smithi079.stderr: 1700K .......... .......... .......... .......... .......... 2% 21.6M 2s 2019-12-11T16:46:06.369 INFO:teuthology.orchestra.run.smithi079.stderr: 1750K .......... .......... .......... .......... .......... 2% 91.3M 2s 2019-12-11T16:46:06.369 INFO:teuthology.orchestra.run.smithi079.stderr: 1800K .......... .......... .......... .......... .......... 2% 246M 2s 2019-12-11T16:46:06.369 INFO:teuthology.orchestra.run.smithi079.stderr: 1850K .......... .......... .......... .......... .......... 2% 260M 2s 2019-12-11T16:46:06.369 INFO:teuthology.orchestra.run.smithi079.stderr: 1900K .......... .......... .......... .......... .......... 2% 239M 2s 2019-12-11T16:46:06.370 INFO:teuthology.orchestra.run.smithi079.stderr: 1950K .......... .......... .......... .......... .......... 2% 184M 2s 2019-12-11T16:46:06.370 INFO:teuthology.orchestra.run.smithi079.stderr: 2000K .......... .......... .......... .......... .......... 2% 159M 2s 2019-12-11T16:46:06.370 INFO:teuthology.orchestra.run.smithi079.stderr: 2050K .......... .......... .......... .......... .......... 2% 182M 2s 2019-12-11T16:46:06.371 INFO:teuthology.orchestra.run.smithi079.stderr: 2100K .......... .......... .......... .......... .......... 2% 194M 2s 2019-12-11T16:46:06.371 INFO:teuthology.orchestra.run.smithi079.stderr: 2150K .......... .......... .......... .......... .......... 2% 159M 1s 2019-12-11T16:46:06.371 INFO:teuthology.orchestra.run.smithi079.stderr: 2200K .......... .......... .......... .......... .......... 3% 172M 1s 2019-12-11T16:46:06.371 INFO:teuthology.orchestra.run.smithi079.stderr: 2250K .......... .......... .......... .......... .......... 3% 155M 1s 2019-12-11T16:46:06.371 INFO:teuthology.orchestra.run.smithi079.stderr: 2300K .......... .......... .......... .......... .......... 3% 189M 1s 2019-12-11T16:46:06.372 INFO:teuthology.orchestra.run.smithi079.stderr: 2350K .......... .......... .......... .......... .......... 3% 144M 1s 2019-12-11T16:46:06.372 INFO:teuthology.orchestra.run.smithi079.stderr: 2400K .......... .......... .......... .......... .......... 3% 172M 1s 2019-12-11T16:46:06.372 INFO:teuthology.orchestra.run.smithi079.stderr: 2450K .......... .......... .......... .......... .......... 3% 167M 1s 2019-12-11T16:46:06.372 INFO:teuthology.orchestra.run.smithi079.stderr: 2500K .......... .......... .......... .......... .......... 3% 182M 1s 2019-12-11T16:46:06.372 INFO:teuthology.orchestra.run.smithi079.stderr: 2550K .......... .......... .......... .......... .......... 3% 163M 1s 2019-12-11T16:46:06.373 INFO:teuthology.orchestra.run.smithi079.stderr: 2600K .......... .......... .......... .......... .......... 3% 167M 1s 2019-12-11T16:46:06.373 INFO:teuthology.orchestra.run.smithi079.stderr: 2650K .......... .......... .......... .......... .......... 3% 166M 1s 2019-12-11T16:46:06.373 INFO:teuthology.orchestra.run.smithi079.stderr: 2700K .......... .......... .......... .......... .......... 3% 155M 1s 2019-12-11T16:46:06.373 INFO:teuthology.orchestra.run.smithi079.stderr: 2750K .......... .......... .......... .......... .......... 3% 132M 1s 2019-12-11T16:46:06.374 INFO:teuthology.orchestra.run.smithi079.stderr: 2800K .......... .......... .......... .......... .......... 3% 179M 1s 2019-12-11T16:46:06.374 INFO:teuthology.orchestra.run.smithi079.stderr: 2850K .......... .......... .......... .......... .......... 3% 175M 1s 2019-12-11T16:46:06.374 INFO:teuthology.orchestra.run.smithi079.stderr: 2900K .......... .......... .......... .......... .......... 4% 190M 1s 2019-12-11T16:46:06.374 INFO:teuthology.orchestra.run.smithi079.stderr: 2950K .......... .......... .......... .......... .......... 4% 193M 1s 2019-12-11T16:46:06.375 INFO:teuthology.orchestra.run.smithi079.stderr: 3000K .......... .......... .......... .......... .......... 4% 174M 1s 2019-12-11T16:46:06.375 INFO:teuthology.orchestra.run.smithi079.stderr: 3050K .......... .......... .......... .......... .......... 4% 212M 1s 2019-12-11T16:46:06.375 INFO:teuthology.orchestra.run.smithi079.stderr: 3100K .......... .......... .......... .......... .......... 4% 213M 1s 2019-12-11T16:46:06.375 INFO:teuthology.orchestra.run.smithi079.stderr: 3150K .......... .......... .......... .......... .......... 4% 130M 1s 2019-12-11T16:46:06.375 INFO:teuthology.orchestra.run.smithi079.stderr: 3200K .......... .......... .......... .......... .......... 4% 166M 1s 2019-12-11T16:46:06.376 INFO:teuthology.orchestra.run.smithi079.stderr: 3250K .......... .......... .......... .......... .......... 4% 177M 1s 2019-12-11T16:46:06.376 INFO:teuthology.orchestra.run.smithi079.stderr: 3300K .......... .......... .......... .......... .......... 4% 156M 1s 2019-12-11T16:46:06.376 INFO:teuthology.orchestra.run.smithi079.stderr: 3350K .......... .......... .......... .......... .......... 4% 135M 1s 2019-12-11T16:46:06.377 INFO:teuthology.orchestra.run.smithi079.stderr: 3400K .......... .......... .......... .......... .......... 4% 149M 1s 2019-12-11T16:46:06.377 INFO:teuthology.orchestra.run.smithi079.stderr: 3450K .......... .......... .......... .......... .......... 4% 138M 1s 2019-12-11T16:46:06.377 INFO:teuthology.orchestra.run.smithi079.stderr: 3500K .......... .......... .......... .......... .......... 4% 145M 1s 2019-12-11T16:46:06.377 INFO:teuthology.orchestra.run.smithi079.stderr: 3550K .......... .......... .......... .......... .......... 4% 132M 1s 2019-12-11T16:46:06.378 INFO:teuthology.orchestra.run.smithi079.stderr: 3600K .......... .......... .......... .......... .......... 4% 148M 1s 2019-12-11T16:46:06.378 INFO:teuthology.orchestra.run.smithi079.stderr: 3650K .......... .......... .......... .......... .......... 5% 161M 1s 2019-12-11T16:46:06.378 INFO:teuthology.orchestra.run.smithi079.stderr: 3700K .......... .......... .......... .......... .......... 5% 151M 1s 2019-12-11T16:46:06.378 INFO:teuthology.orchestra.run.smithi079.stderr: 3750K .......... .......... .......... .......... .......... 5% 139M 1s 2019-12-11T16:46:06.378 INFO:teuthology.orchestra.run.smithi079.stderr: 3800K .......... .......... .......... .......... .......... 5% 157M 1s 2019-12-11T16:46:06.379 INFO:teuthology.orchestra.run.smithi079.stderr: 3850K .......... .......... .......... .......... .......... 5% 176M 1s 2019-12-11T16:46:06.379 INFO:teuthology.orchestra.run.smithi079.stderr: 3900K .......... .......... .......... .......... .......... 5% 115M 1s 2019-12-11T16:46:06.379 INFO:teuthology.orchestra.run.smithi079.stderr: 3950K .......... .......... .......... .......... .......... 5% 108M 1s 2019-12-11T16:46:06.379 INFO:teuthology.orchestra.run.smithi079.stderr: 4000K .......... .......... .......... .......... .......... 5% 151M 1s 2019-12-11T16:46:06.379 INFO:teuthology.orchestra.run.smithi079.stderr: 4050K .......... .......... .......... .......... .......... 5% 177M 1s 2019-12-11T16:46:06.380 INFO:teuthology.orchestra.run.smithi079.stderr: 4100K .......... .......... .......... .......... .......... 5% 227M 1s 2019-12-11T16:46:06.380 INFO:teuthology.orchestra.run.smithi079.stderr: 4150K .......... .......... .......... .......... .......... 5% 201M 1s 2019-12-11T16:46:06.380 INFO:teuthology.orchestra.run.smithi079.stderr: 4200K .......... .......... .......... .......... .......... 5% 161M 1s 2019-12-11T16:46:06.380 INFO:teuthology.orchestra.run.smithi079.stderr: 4250K .......... .......... .......... .......... .......... 5% 197M 1s 2019-12-11T16:46:06.381 INFO:teuthology.orchestra.run.smithi079.stderr: 4300K .......... .......... .......... .......... .......... 5% 166M 1s 2019-12-11T16:46:06.381 INFO:teuthology.orchestra.run.smithi079.stderr: 4350K .......... .......... .......... .......... .......... 5% 165M 1s 2019-12-11T16:46:06.381 INFO:teuthology.orchestra.run.smithi079.stderr: 4400K .......... .......... .......... .......... .......... 6% 186M 1s 2019-12-11T16:46:06.381 INFO:teuthology.orchestra.run.smithi079.stderr: 4450K .......... .......... .......... .......... .......... 6% 178M 1s 2019-12-11T16:46:06.381 INFO:teuthology.orchestra.run.smithi079.stderr: 4500K .......... .......... .......... .......... .......... 6% 156M 1s 2019-12-11T16:46:06.382 INFO:teuthology.orchestra.run.smithi079.stderr: 4550K .......... .......... .......... .......... .......... 6% 155M 1s 2019-12-11T16:46:06.382 INFO:teuthology.orchestra.run.smithi079.stderr: 4600K .......... .......... .......... .......... .......... 6% 167M 1s 2019-12-11T16:46:06.382 INFO:teuthology.orchestra.run.smithi079.stderr: 4650K .......... .......... .......... .......... .......... 6% 213M 1s 2019-12-11T16:46:06.382 INFO:teuthology.orchestra.run.smithi079.stderr: 4700K .......... .......... .......... .......... .......... 6% 181M 1s 2019-12-11T16:46:06.383 INFO:teuthology.orchestra.run.smithi079.stderr: 4750K .......... .......... .......... .......... .......... 6% 136M 1s 2019-12-11T16:46:06.383 INFO:teuthology.orchestra.run.smithi079.stderr: 4800K .......... .......... .......... .......... .......... 6% 185M 1s 2019-12-11T16:46:06.383 INFO:teuthology.orchestra.run.smithi079.stderr: 4850K .......... .......... .......... .......... .......... 6% 183M 1s 2019-12-11T16:46:06.383 INFO:teuthology.orchestra.run.smithi079.stderr: 4900K .......... .......... .......... .......... .......... 6% 197M 1s 2019-12-11T16:46:06.384 INFO:teuthology.orchestra.run.smithi079.stderr: 4950K .......... .......... .......... .......... .......... 6% 140M 1s 2019-12-11T16:46:06.384 INFO:teuthology.orchestra.run.smithi079.stderr: 5000K .......... .......... .......... .......... .......... 6% 205M 1s 2019-12-11T16:46:06.384 INFO:teuthology.orchestra.run.smithi079.stderr: 5050K .......... .......... .......... .......... .......... 6% 130M 1s 2019-12-11T16:46:06.384 INFO:teuthology.orchestra.run.smithi079.stderr: 5100K .......... .......... .......... .......... .......... 7% 191M 1s 2019-12-11T16:46:06.384 INFO:teuthology.orchestra.run.smithi079.stderr: 5150K .......... .......... .......... .......... .......... 7% 149M 1s 2019-12-11T16:46:06.385 INFO:teuthology.orchestra.run.smithi079.stderr: 5200K .......... .......... .......... .......... .......... 7% 164M 1s 2019-12-11T16:46:06.385 INFO:teuthology.orchestra.run.smithi079.stderr: 5250K .......... .......... .......... .......... .......... 7% 168M 1s 2019-12-11T16:46:06.385 INFO:teuthology.orchestra.run.smithi079.stderr: 5300K .......... .......... .......... .......... .......... 7% 168M 1s 2019-12-11T16:46:06.385 INFO:teuthology.orchestra.run.smithi079.stderr: 5350K .......... .......... .......... .......... .......... 7% 210M 1s 2019-12-11T16:46:06.386 INFO:teuthology.orchestra.run.smithi079.stderr: 5400K .......... .......... .......... .......... .......... 7% 178M 1s 2019-12-11T16:46:06.386 INFO:teuthology.orchestra.run.smithi079.stderr: 5450K .......... .......... .......... .......... .......... 7% 235M 1s 2019-12-11T16:46:06.386 INFO:teuthology.orchestra.run.smithi079.stderr: 5500K .......... .......... .......... .......... .......... 7% 198M 1s 2019-12-11T16:46:06.386 INFO:teuthology.orchestra.run.smithi079.stderr: 5550K .......... .......... .......... .......... .......... 7% 160M 1s 2019-12-11T16:46:06.387 INFO:teuthology.orchestra.run.smithi079.stderr: 5600K .......... .......... .......... .......... .......... 7% 190M 1s 2019-12-11T16:46:06.387 INFO:teuthology.orchestra.run.smithi079.stderr: 5650K .......... .......... .......... .......... .......... 7% 166M 1s 2019-12-11T16:46:06.387 INFO:teuthology.orchestra.run.smithi079.stderr: 5700K .......... .......... .......... .......... .......... 7% 185M 1s 2019-12-11T16:46:06.387 INFO:teuthology.orchestra.run.smithi079.stderr: 5750K .......... .......... .......... .......... .......... 7% 165M 1s 2019-12-11T16:46:06.388 INFO:teuthology.orchestra.run.smithi079.stderr: 5800K .......... .......... .......... .......... .......... 7% 190M 1s 2019-12-11T16:46:06.388 INFO:teuthology.orchestra.run.smithi079.stderr: 5850K .......... .......... .......... .......... .......... 8% 185M 1s 2019-12-11T16:46:06.388 INFO:teuthology.orchestra.run.smithi079.stderr: 5900K .......... .......... .......... .......... .......... 8% 184M 1s 2019-12-11T16:46:06.388 INFO:teuthology.orchestra.run.smithi079.stderr: 5950K .......... .......... .......... .......... .......... 8% 148M 1s 2019-12-11T16:46:06.389 INFO:teuthology.orchestra.run.smithi079.stderr: 6000K .......... .......... .......... .......... .......... 8% 214M 1s 2019-12-11T16:46:06.389 INFO:teuthology.orchestra.run.smithi079.stderr: 6050K .......... .......... .......... .......... .......... 8% 168M 1s 2019-12-11T16:46:06.389 INFO:teuthology.orchestra.run.smithi079.stderr: 6100K .......... .......... .......... .......... .......... 8% 178M 1s 2019-12-11T16:46:06.389 INFO:teuthology.orchestra.run.smithi079.stderr: 6150K .......... .......... .......... .......... .......... 8% 143M 1s 2019-12-11T16:46:06.389 INFO:teuthology.orchestra.run.smithi079.stderr: 6200K .......... .......... .......... .......... .......... 8% 198M 1s 2019-12-11T16:46:06.390 INFO:teuthology.orchestra.run.smithi079.stderr: 6250K .......... .......... .......... .......... .......... 8% 187M 1s 2019-12-11T16:46:06.390 INFO:teuthology.orchestra.run.smithi079.stderr: 6300K .......... .......... .......... .......... .......... 8% 199M 1s 2019-12-11T16:46:06.390 INFO:teuthology.orchestra.run.smithi079.stderr: 6350K .......... .......... .......... .......... .......... 8% 141M 1s 2019-12-11T16:46:06.390 INFO:teuthology.orchestra.run.smithi079.stderr: 6400K .......... .......... .......... .......... .......... 8% 150M 1s 2019-12-11T16:46:06.390 INFO:teuthology.orchestra.run.smithi079.stderr: 6450K .......... .......... .......... .......... .......... 8% 231M 1s 2019-12-11T16:46:06.390 INFO:teuthology.orchestra.run.smithi079.stderr: 6500K .......... .......... .......... .......... .......... 8% 191M 1s 2019-12-11T16:46:06.390 INFO:teuthology.orchestra.run.smithi079.stderr: 6550K .......... .......... .......... .......... .......... 8% 144M 1s 2019-12-11T16:46:06.391 INFO:teuthology.orchestra.run.smithi079.stderr: 6600K .......... .......... .......... .......... .......... 9% 174M 1s 2019-12-11T16:46:06.391 INFO:teuthology.orchestra.run.smithi079.stderr: 6650K .......... .......... .......... .......... .......... 9% 232M 1s 2019-12-11T16:46:06.391 INFO:teuthology.orchestra.run.smithi079.stderr: 6700K .......... .......... .......... .......... .......... 9% 170M 1s 2019-12-11T16:46:06.391 INFO:teuthology.orchestra.run.smithi079.stderr: 6750K .......... .......... .......... .......... .......... 9% 169M 1s 2019-12-11T16:46:06.391 INFO:teuthology.orchestra.run.smithi079.stderr: 6800K .......... .......... .......... .......... .......... 9% 242M 1s 2019-12-11T16:46:06.391 INFO:teuthology.orchestra.run.smithi079.stderr: 6850K .......... .......... .......... .......... .......... 9% 204M 1s 2019-12-11T16:46:06.392 INFO:teuthology.orchestra.run.smithi079.stderr: 6900K .......... .......... .......... .......... .......... 9% 183M 1s 2019-12-11T16:46:06.392 INFO:teuthology.orchestra.run.smithi079.stderr: 6950K .......... .......... .......... .......... .......... 9% 208M 1s 2019-12-11T16:46:06.392 INFO:teuthology.orchestra.run.smithi079.stderr: 7000K .......... .......... .......... .......... .......... 9% 191M 1s 2019-12-11T16:46:06.392 INFO:teuthology.orchestra.run.smithi079.stderr: 7050K .......... .......... .......... .......... .......... 9% 184M 1s 2019-12-11T16:46:06.392 INFO:teuthology.orchestra.run.smithi079.stderr: 7100K .......... .......... .......... .......... .......... 9% 189M 1s 2019-12-11T16:46:06.392 INFO:teuthology.orchestra.run.smithi079.stderr: 7150K .......... .......... .......... .......... .......... 9% 168M 1s 2019-12-11T16:46:06.393 INFO:teuthology.orchestra.run.smithi079.stderr: 7200K .......... .......... .......... .......... .......... 9% 215M 1s 2019-12-11T16:46:06.393 INFO:teuthology.orchestra.run.smithi079.stderr: 7250K .......... .......... .......... .......... .......... 9% 174M 1s 2019-12-11T16:46:06.393 INFO:teuthology.orchestra.run.smithi079.stderr: 7300K .......... .......... .......... .......... .......... 9% 189M 1s 2019-12-11T16:46:06.393 INFO:teuthology.orchestra.run.smithi079.stderr: 7350K .......... .......... .......... .......... .......... 10% 165M 1s 2019-12-11T16:46:06.393 INFO:teuthology.orchestra.run.smithi079.stderr: 7400K .......... .......... .......... .......... .......... 10% 245M 1s 2019-12-11T16:46:06.393 INFO:teuthology.orchestra.run.smithi079.stderr: 7450K .......... .......... .......... .......... .......... 10% 169M 1s 2019-12-11T16:46:06.393 INFO:teuthology.orchestra.run.smithi079.stderr: 7500K .......... .......... .......... .......... .......... 10% 194M 1s 2019-12-11T16:46:06.394 INFO:teuthology.orchestra.run.smithi079.stderr: 7550K .......... .......... .......... .......... .......... 10% 173M 1s 2019-12-11T16:46:06.394 INFO:teuthology.orchestra.run.smithi079.stderr: 7600K .......... .......... .......... .......... .......... 10% 198M 1s 2019-12-11T16:46:06.394 INFO:teuthology.orchestra.run.smithi079.stderr: 7650K .......... .......... .......... .......... .......... 10% 173M 1s 2019-12-11T16:46:06.394 INFO:teuthology.orchestra.run.smithi079.stderr: 7700K .......... .......... .......... .......... .......... 10% 240M 1s 2019-12-11T16:46:06.394 INFO:teuthology.orchestra.run.smithi079.stderr: 7750K .......... .......... .......... .......... .......... 10% 151M 1s 2019-12-11T16:46:06.394 INFO:teuthology.orchestra.run.smithi079.stderr: 7800K .......... .......... .......... .......... .......... 10% 242M 1s 2019-12-11T16:46:06.394 INFO:teuthology.orchestra.run.smithi079.stderr: 7850K .......... .......... .......... .......... .......... 10% 155M 1s 2019-12-11T16:46:06.395 INFO:teuthology.orchestra.run.smithi079.stderr: 7900K .......... .......... .......... .......... .......... 10% 242M 1s 2019-12-11T16:46:06.395 INFO:teuthology.orchestra.run.smithi079.stderr: 7950K .......... .......... .......... .......... .......... 10% 164M 1s 2019-12-11T16:46:06.395 INFO:teuthology.orchestra.run.smithi079.stderr: 8000K .......... .......... .......... .......... .......... 10% 166M 1s 2019-12-11T16:46:06.395 INFO:teuthology.orchestra.run.smithi079.stderr: 8050K .......... .......... .......... .......... .......... 11% 166M 1s 2019-12-11T16:46:06.395 INFO:teuthology.orchestra.run.smithi079.stderr: 8100K .......... .......... .......... .......... .......... 11% 240M 1s 2019-12-11T16:46:06.395 INFO:teuthology.orchestra.run.smithi079.stderr: 8150K .......... .......... .......... .......... .......... 11% 185M 1s 2019-12-11T16:46:06.396 INFO:teuthology.orchestra.run.smithi079.stderr: 8200K .......... .......... .......... .......... .......... 11% 209M 1s 2019-12-11T16:46:06.396 INFO:teuthology.orchestra.run.smithi079.stderr: 8250K .......... .......... .......... .......... .......... 11% 235M 1s 2019-12-11T16:46:06.396 INFO:teuthology.orchestra.run.smithi079.stderr: 8300K .......... .......... .......... .......... .......... 11% 231M 1s 2019-12-11T16:46:06.396 INFO:teuthology.orchestra.run.smithi079.stderr: 8350K .......... .......... .......... .......... .......... 11% 139M 1s 2019-12-11T16:46:06.396 INFO:teuthology.orchestra.run.smithi079.stderr: 8400K .......... .......... .......... .......... .......... 11% 221M 1s 2019-12-11T16:46:06.396 INFO:teuthology.orchestra.run.smithi079.stderr: 8450K .......... .......... .......... .......... .......... 11% 167M 1s 2019-12-11T16:46:06.397 INFO:teuthology.orchestra.run.smithi079.stderr: 8500K .......... .......... .......... .......... .......... 11% 210M 1s 2019-12-11T16:46:06.397 INFO:teuthology.orchestra.run.smithi079.stderr: 8550K .......... .......... .......... .......... .......... 11% 165M 1s 2019-12-11T16:46:06.397 INFO:teuthology.orchestra.run.smithi079.stderr: 8600K .......... .......... .......... .......... .......... 11% 234M 1s 2019-12-11T16:46:06.397 INFO:teuthology.orchestra.run.smithi079.stderr: 8650K .......... .......... .......... .......... .......... 11% 190M 1s 2019-12-11T16:46:06.397 INFO:teuthology.orchestra.run.smithi079.stderr: 8700K .......... .......... .......... .......... .......... 11% 175M 1s 2019-12-11T16:46:06.397 INFO:teuthology.orchestra.run.smithi079.stderr: 8750K .......... .......... .......... .......... .......... 11% 162M 1s 2019-12-11T16:46:06.398 INFO:teuthology.orchestra.run.smithi079.stderr: 8800K .......... .......... .......... .......... .......... 12% 196M 1s 2019-12-11T16:46:06.398 INFO:teuthology.orchestra.run.smithi079.stderr: 8850K .......... .......... .......... .......... .......... 12% 159M 1s 2019-12-11T16:46:06.398 INFO:teuthology.orchestra.run.smithi079.stderr: 8900K .......... .......... .......... .......... .......... 12% 249M 1s 2019-12-11T16:46:06.398 INFO:teuthology.orchestra.run.smithi079.stderr: 8950K .......... .......... .......... .......... .......... 12% 175M 1s 2019-12-11T16:46:06.398 INFO:teuthology.orchestra.run.smithi079.stderr: 9000K .......... .......... .......... .......... .......... 12% 160M 1s 2019-12-11T16:46:06.398 INFO:teuthology.orchestra.run.smithi079.stderr: 9050K .......... .......... .......... .......... .......... 12% 221M 1s 2019-12-11T16:46:06.399 INFO:teuthology.orchestra.run.smithi079.stderr: 9100K .......... .......... .......... .......... .......... 12% 191M 1s 2019-12-11T16:46:06.399 INFO:teuthology.orchestra.run.smithi079.stderr: 9150K .......... .......... .......... .......... .......... 12% 209M 1s 2019-12-11T16:46:06.399 INFO:teuthology.orchestra.run.smithi079.stderr: 9200K .......... .......... .......... .......... .......... 12% 159M 1s 2019-12-11T16:46:06.399 INFO:teuthology.orchestra.run.smithi079.stderr: 9250K .......... .......... .......... .......... .......... 12% 248M 1s 2019-12-11T16:46:06.399 INFO:teuthology.orchestra.run.smithi079.stderr: 9300K .......... .......... .......... .......... .......... 12% 177M 1s 2019-12-11T16:46:06.399 INFO:teuthology.orchestra.run.smithi079.stderr: 9350K .......... .......... .......... .......... .......... 12% 220M 1s 2019-12-11T16:46:06.400 INFO:teuthology.orchestra.run.smithi079.stderr: 9400K .......... .......... .......... .......... .......... 12% 180M 1s 2019-12-11T16:46:06.400 INFO:teuthology.orchestra.run.smithi079.stderr: 9450K .......... .......... .......... .......... .......... 12% 188M 1s 2019-12-11T16:46:06.400 INFO:teuthology.orchestra.run.smithi079.stderr: 9500K .......... .......... .......... .......... .......... 12% 190M 1s 2019-12-11T16:46:06.400 INFO:teuthology.orchestra.run.smithi079.stderr: 9550K .......... .......... .......... .......... .......... 13% 207M 1s 2019-12-11T16:46:06.400 INFO:teuthology.orchestra.run.smithi079.stderr: 9600K .......... .......... .......... .......... .......... 13% 181M 1s 2019-12-11T16:46:06.401 INFO:teuthology.orchestra.run.smithi079.stderr: 9650K .......... .......... .......... .......... .......... 13% 222M 1s 2019-12-11T16:46:06.401 INFO:teuthology.orchestra.run.smithi079.stderr: 9700K .......... .......... .......... .......... .......... 13% 246M 1s 2019-12-11T16:46:06.401 INFO:teuthology.orchestra.run.smithi079.stderr: 9750K .......... .......... .......... .......... .......... 13% 221M 1s 2019-12-11T16:46:06.401 INFO:teuthology.orchestra.run.smithi079.stderr: 9800K .......... .......... .......... .......... .......... 13% 210M 1s 2019-12-11T16:46:06.401 INFO:teuthology.orchestra.run.smithi079.stderr: 9850K .......... .......... .......... .......... .......... 13% 169M 1s 2019-12-11T16:46:06.401 INFO:teuthology.orchestra.run.smithi079.stderr: 9900K .......... .......... .......... .......... .......... 13% 214M 1s 2019-12-11T16:46:06.402 INFO:teuthology.orchestra.run.smithi079.stderr: 9950K .......... .......... .......... .......... .......... 13% 174M 1s 2019-12-11T16:46:06.402 INFO:teuthology.orchestra.run.smithi079.stderr: 10000K .......... .......... .......... .......... .......... 13% 178M 1s 2019-12-11T16:46:06.402 INFO:teuthology.orchestra.run.smithi079.stderr: 10050K .......... .......... .......... .......... .......... 13% 231M 1s 2019-12-11T16:46:06.402 INFO:teuthology.orchestra.run.smithi079.stderr: 10100K .......... .......... .......... .......... .......... 13% 188M 1s 2019-12-11T16:46:06.402 INFO:teuthology.orchestra.run.smithi079.stderr: 10150K .......... .......... .......... .......... .......... 13% 191M 1s 2019-12-11T16:46:06.402 INFO:teuthology.orchestra.run.smithi079.stderr: 10200K .......... .......... .......... .......... .......... 13% 179M 1s 2019-12-11T16:46:06.402 INFO:teuthology.orchestra.run.smithi079.stderr: 10250K .......... .......... .......... .......... .......... 14% 239M 1s 2019-12-11T16:46:06.403 INFO:teuthology.orchestra.run.smithi079.stderr: 10300K .......... .......... .......... .......... .......... 14% 228M 1s 2019-12-11T16:46:06.403 INFO:teuthology.orchestra.run.smithi079.stderr: 10350K .......... .......... .......... .......... .......... 14% 177M 1s 2019-12-11T16:46:06.403 INFO:teuthology.orchestra.run.smithi079.stderr: 10400K .......... .......... .......... .......... .......... 14% 243M 1s 2019-12-11T16:46:06.403 INFO:teuthology.orchestra.run.smithi079.stderr: 10450K .......... .......... .......... .......... .......... 14% 186M 1s 2019-12-11T16:46:06.403 INFO:teuthology.orchestra.run.smithi079.stderr: 10500K .......... .......... .......... .......... .......... 14% 234M 1s 2019-12-11T16:46:06.403 INFO:teuthology.orchestra.run.smithi079.stderr: 10550K .......... .......... .......... .......... .......... 14% 193M 1s 2019-12-11T16:46:06.404 INFO:teuthology.orchestra.run.smithi079.stderr: 10600K .......... .......... .......... .......... .......... 14% 192M 1s 2019-12-11T16:46:06.404 INFO:teuthology.orchestra.run.smithi079.stderr: 10650K .......... .......... .......... .......... .......... 14% 208M 1s 2019-12-11T16:46:06.404 INFO:teuthology.orchestra.run.smithi079.stderr: 10700K .......... .......... .......... .......... .......... 14% 240M 1s 2019-12-11T16:46:06.404 INFO:teuthology.orchestra.run.smithi079.stderr: 10750K .......... .......... .......... .......... .......... 14% 170M 1s 2019-12-11T16:46:06.404 INFO:teuthology.orchestra.run.smithi079.stderr: 10800K .......... .......... .......... .......... .......... 14% 256M 1s 2019-12-11T16:46:06.404 INFO:teuthology.orchestra.run.smithi079.stderr: 10850K .......... .......... .......... .......... .......... 14% 183M 1s 2019-12-11T16:46:06.404 INFO:teuthology.orchestra.run.smithi079.stderr: 10900K .......... .......... .......... .......... .......... 14% 260M 1s 2019-12-11T16:46:06.405 INFO:teuthology.orchestra.run.smithi079.stderr: 10950K .......... .......... .......... .......... .......... 14% 180M 1s 2019-12-11T16:46:06.405 INFO:teuthology.orchestra.run.smithi079.stderr: 11000K .......... .......... .......... .......... .......... 15% 228M 1s 2019-12-11T16:46:06.405 INFO:teuthology.orchestra.run.smithi079.stderr: 11050K .......... .......... .......... .......... .......... 15% 206M 1s 2019-12-11T16:46:06.405 INFO:teuthology.orchestra.run.smithi079.stderr: 11100K .......... .......... .......... .......... .......... 15% 256M 1s 2019-12-11T16:46:06.405 INFO:teuthology.orchestra.run.smithi079.stderr: 11150K .......... .......... .......... .......... .......... 15% 232M 1s 2019-12-11T16:46:06.405 INFO:teuthology.orchestra.run.smithi079.stderr: 11200K .......... .......... .......... .......... .......... 15% 209M 1s 2019-12-11T16:46:06.406 INFO:teuthology.orchestra.run.smithi079.stderr: 11250K .......... .......... .......... .......... .......... 15% 205M 1s 2019-12-11T16:46:06.406 INFO:teuthology.orchestra.run.smithi079.stderr: 11300K .......... .......... .......... .......... .......... 15% 217M 1s 2019-12-11T16:46:06.406 INFO:teuthology.orchestra.run.smithi079.stderr: 11350K .......... .......... .......... .......... .......... 15% 204M 1s 2019-12-11T16:46:06.406 INFO:teuthology.orchestra.run.smithi079.stderr: 11400K .......... .......... .......... .......... .......... 15% 196M 1s 2019-12-11T16:46:06.406 INFO:teuthology.orchestra.run.smithi079.stderr: 11450K .......... .......... .......... .......... .......... 15% 217M 1s 2019-12-11T16:46:06.406 INFO:teuthology.orchestra.run.smithi079.stderr: 11500K .......... .......... .......... .......... .......... 15% 195M 1s 2019-12-11T16:46:06.406 INFO:teuthology.orchestra.run.smithi079.stderr: 11550K .......... .......... .......... .......... .......... 15% 182M 1s 2019-12-11T16:46:06.407 INFO:teuthology.orchestra.run.smithi079.stderr: 11600K .......... .......... .......... .......... .......... 15% 224M 1s 2019-12-11T16:46:06.407 INFO:teuthology.orchestra.run.smithi079.stderr: 11650K .......... .......... .......... .......... .......... 15% 227M 1s 2019-12-11T16:46:06.407 INFO:teuthology.orchestra.run.smithi079.stderr: 11700K .......... .......... .......... .......... .......... 15% 204M 1s 2019-12-11T16:46:06.407 INFO:teuthology.orchestra.run.smithi079.stderr: 11750K .......... .......... .......... .......... .......... 16% 224M 1s 2019-12-11T16:46:06.407 INFO:teuthology.orchestra.run.smithi079.stderr: 11800K .......... .......... .......... .......... .......... 16% 230M 1s 2019-12-11T16:46:06.407 INFO:teuthology.orchestra.run.smithi079.stderr: 11850K .......... .......... .......... .......... .......... 16% 232M 1s 2019-12-11T16:46:06.407 INFO:teuthology.orchestra.run.smithi079.stderr: 11900K .......... .......... .......... .......... .......... 16% 241M 1s 2019-12-11T16:46:06.408 INFO:teuthology.orchestra.run.smithi079.stderr: 11950K .......... .......... .......... .......... .......... 16% 169M 1s 2019-12-11T16:46:06.408 INFO:teuthology.orchestra.run.smithi079.stderr: 12000K .......... .......... .......... .......... .......... 16% 263M 1s 2019-12-11T16:46:06.408 INFO:teuthology.orchestra.run.smithi079.stderr: 12050K .......... .......... .......... .......... .......... 16% 216M 1s 2019-12-11T16:46:06.408 INFO:teuthology.orchestra.run.smithi079.stderr: 12100K .......... .......... .......... .......... .......... 16% 200M 1s 2019-12-11T16:46:06.408 INFO:teuthology.orchestra.run.smithi079.stderr: 12150K .......... .......... .......... .......... .......... 16% 181M 0s 2019-12-11T16:46:06.408 INFO:teuthology.orchestra.run.smithi079.stderr: 12200K .......... .......... .......... .......... .......... 16% 198M 0s 2019-12-11T16:46:06.409 INFO:teuthology.orchestra.run.smithi079.stderr: 12250K .......... .......... .......... .......... .......... 16% 237M 0s 2019-12-11T16:46:06.409 INFO:teuthology.orchestra.run.smithi079.stderr: 12300K .......... .......... .......... .......... .......... 16% 213M 0s 2019-12-11T16:46:06.409 INFO:teuthology.orchestra.run.smithi079.stderr: 12350K .......... .......... .......... .......... .......... 16% 170M 0s 2019-12-11T16:46:06.409 INFO:teuthology.orchestra.run.smithi079.stderr: 12400K .......... .......... .......... .......... .......... 16% 199M 0s 2019-12-11T16:46:06.409 INFO:teuthology.orchestra.run.smithi079.stderr: 12450K .......... .......... .......... .......... .......... 16% 267M 0s 2019-12-11T16:46:06.409 INFO:teuthology.orchestra.run.smithi079.stderr: 12500K .......... .......... .......... .......... .......... 17% 192M 0s 2019-12-11T16:46:06.410 INFO:teuthology.orchestra.run.smithi079.stderr: 12550K .......... .......... .......... .......... .......... 17% 165M 0s 2019-12-11T16:46:06.410 INFO:teuthology.orchestra.run.smithi079.stderr: 12600K .......... .......... .......... .......... .......... 17% 236M 0s 2019-12-11T16:46:06.410 INFO:teuthology.orchestra.run.smithi079.stderr: 12650K .......... .......... .......... .......... .......... 17% 203M 0s 2019-12-11T16:46:06.410 INFO:teuthology.orchestra.run.smithi079.stderr: 12700K .......... .......... .......... .......... .......... 17% 232M 0s 2019-12-11T16:46:06.410 INFO:teuthology.orchestra.run.smithi079.stderr: 12750K .......... .......... .......... .......... .......... 17% 206M 0s 2019-12-11T16:46:06.410 INFO:teuthology.orchestra.run.smithi079.stderr: 12800K .......... .......... .......... .......... .......... 17% 188M 0s 2019-12-11T16:46:06.410 INFO:teuthology.orchestra.run.smithi079.stderr: 12850K .......... .......... .......... .......... .......... 17% 265M 0s 2019-12-11T16:46:06.411 INFO:teuthology.orchestra.run.smithi079.stderr: 12900K .......... .......... .......... .......... .......... 17% 337M 0s 2019-12-11T16:46:06.411 INFO:teuthology.orchestra.run.smithi079.stderr: 12950K .......... .......... .......... .......... .......... 17% 280M 0s 2019-12-11T16:46:06.411 INFO:teuthology.orchestra.run.smithi079.stderr: 13000K .......... .......... .......... .......... .......... 17% 339M 0s 2019-12-11T16:46:06.411 INFO:teuthology.orchestra.run.smithi079.stderr: 13050K .......... .......... .......... .......... .......... 17% 321M 0s 2019-12-11T16:46:06.411 INFO:teuthology.orchestra.run.smithi079.stderr: 13100K .......... .......... .......... .......... .......... 17% 323M 0s 2019-12-11T16:46:06.412 INFO:teuthology.orchestra.run.smithi079.stderr: 13150K .......... .......... .......... .......... .......... 17% 255M 0s 2019-12-11T16:46:06.412 INFO:teuthology.orchestra.run.smithi079.stderr: 13200K .......... .......... .......... .......... .......... 18% 222M 0s 2019-12-11T16:46:06.412 INFO:teuthology.orchestra.run.smithi079.stderr: 13250K .......... .......... .......... .......... .......... 18% 205M 0s 2019-12-11T16:46:06.412 INFO:teuthology.orchestra.run.smithi079.stderr: 13300K .......... .......... .......... .......... .......... 18% 238M 0s 2019-12-11T16:46:06.412 INFO:teuthology.orchestra.run.smithi079.stderr: 13350K .......... .......... .......... .......... .......... 18% 213M 0s 2019-12-11T16:46:06.412 INFO:teuthology.orchestra.run.smithi079.stderr: 13400K .......... .......... .......... .......... .......... 18% 264M 0s 2019-12-11T16:46:06.412 INFO:teuthology.orchestra.run.smithi079.stderr: 13450K .......... .......... .......... .......... .......... 18% 187M 0s 2019-12-11T16:46:06.413 INFO:teuthology.orchestra.run.smithi079.stderr: 13500K .......... .......... .......... .......... .......... 18% 188M 0s 2019-12-11T16:46:06.413 INFO:teuthology.orchestra.run.smithi079.stderr: 13550K .......... .......... .......... .......... .......... 18% 194M 0s 2019-12-11T16:46:06.413 INFO:teuthology.orchestra.run.smithi079.stderr: 13600K .......... .......... .......... .......... .......... 18% 235M 0s 2019-12-11T16:46:06.413 INFO:teuthology.orchestra.run.smithi079.stderr: 13650K .......... .......... .......... .......... .......... 18% 205M 0s 2019-12-11T16:46:06.413 INFO:teuthology.orchestra.run.smithi079.stderr: 13700K .......... .......... .......... .......... .......... 18% 259M 0s 2019-12-11T16:46:06.413 INFO:teuthology.orchestra.run.smithi079.stderr: 13750K .......... .......... .......... .......... .......... 18% 161M 0s 2019-12-11T16:46:06.413 INFO:teuthology.orchestra.run.smithi079.stderr: 13800K .......... .......... .......... .......... .......... 18% 198M 0s 2019-12-11T16:46:06.414 INFO:teuthology.orchestra.run.smithi079.stderr: 13850K .......... .......... .......... .......... .......... 18% 221M 0s 2019-12-11T16:46:06.414 INFO:teuthology.orchestra.run.smithi079.stderr: 13900K .......... .......... .......... .......... .......... 18% 206M 0s 2019-12-11T16:46:06.414 INFO:teuthology.orchestra.run.smithi079.stderr: 13950K .......... .......... .......... .......... .......... 19% 165M 0s 2019-12-11T16:46:06.414 INFO:teuthology.orchestra.run.smithi079.stderr: 14000K .......... .......... .......... .......... .......... 19% 196M 0s 2019-12-11T16:46:06.414 INFO:teuthology.orchestra.run.smithi079.stderr: 14050K .......... .......... .......... .......... .......... 19% 272M 0s 2019-12-11T16:46:06.414 INFO:teuthology.orchestra.run.smithi079.stderr: 14100K .......... .......... .......... .......... .......... 19% 200M 0s 2019-12-11T16:46:06.414 INFO:teuthology.orchestra.run.smithi079.stderr: 14150K .......... .......... .......... .......... .......... 19% 211M 0s 2019-12-11T16:46:06.415 INFO:teuthology.orchestra.run.smithi079.stderr: 14200K .......... .......... .......... .......... .......... 19% 208M 0s 2019-12-11T16:46:06.415 INFO:teuthology.orchestra.run.smithi079.stderr: 14250K .......... .......... .......... .......... .......... 19% 214M 0s 2019-12-11T16:46:06.415 INFO:teuthology.orchestra.run.smithi079.stderr: 14300K .......... .......... .......... .......... .......... 19% 216M 0s 2019-12-11T16:46:06.415 INFO:teuthology.orchestra.run.smithi079.stderr: 14350K .......... .......... .......... .......... .......... 19% 184M 0s 2019-12-11T16:46:06.415 INFO:teuthology.orchestra.run.smithi079.stderr: 14400K .......... .......... .......... .......... .......... 19% 231M 0s 2019-12-11T16:46:06.415 INFO:teuthology.orchestra.run.smithi079.stderr: 14450K .......... .......... .......... .......... .......... 19% 220M 0s 2019-12-11T16:46:06.415 INFO:teuthology.orchestra.run.smithi079.stderr: 14500K .......... .......... .......... .......... .......... 19% 197M 0s 2019-12-11T16:46:06.416 INFO:teuthology.orchestra.run.smithi079.stderr: 14550K .......... .......... .......... .......... .......... 19% 241M 0s 2019-12-11T16:46:06.416 INFO:teuthology.orchestra.run.smithi079.stderr: 14600K .......... .......... .......... .......... .......... 19% 244M 0s 2019-12-11T16:46:06.416 INFO:teuthology.orchestra.run.smithi079.stderr: 14650K .......... .......... .......... .......... .......... 19% 195M 0s 2019-12-11T16:46:06.416 INFO:teuthology.orchestra.run.smithi079.stderr: 14700K .......... .......... .......... .......... .......... 20% 216M 0s 2019-12-11T16:46:06.416 INFO:teuthology.orchestra.run.smithi079.stderr: 14750K .......... .......... .......... .......... .......... 20% 188M 0s 2019-12-11T16:46:06.416 INFO:teuthology.orchestra.run.smithi079.stderr: 14800K .......... .......... .......... .......... .......... 20% 184M 0s 2019-12-11T16:46:06.417 INFO:teuthology.orchestra.run.smithi079.stderr: 14850K .......... .......... .......... .......... .......... 20% 240M 0s 2019-12-11T16:46:06.417 INFO:teuthology.orchestra.run.smithi079.stderr: 14900K .......... .......... .......... .......... .......... 20% 270M 0s 2019-12-11T16:46:06.417 INFO:teuthology.orchestra.run.smithi079.stderr: 14950K .......... .......... .......... .......... .......... 20% 177M 0s 2019-12-11T16:46:06.417 INFO:teuthology.orchestra.run.smithi079.stderr: 15000K .......... .......... .......... .......... .......... 20% 261M 0s 2019-12-11T16:46:06.417 INFO:teuthology.orchestra.run.smithi079.stderr: 15050K .......... .......... .......... .......... .......... 20% 179M 0s 2019-12-11T16:46:06.417 INFO:teuthology.orchestra.run.smithi079.stderr: 15100K .......... .......... .......... .......... .......... 20% 221M 0s 2019-12-11T16:46:06.417 INFO:teuthology.orchestra.run.smithi079.stderr: 15150K .......... .......... .......... .......... .......... 20% 184M 0s 2019-12-11T16:46:06.418 INFO:teuthology.orchestra.run.smithi079.stderr: 15200K .......... .......... .......... .......... .......... 20% 270M 0s 2019-12-11T16:46:06.418 INFO:teuthology.orchestra.run.smithi079.stderr: 15250K .......... .......... .......... .......... .......... 20% 212M 0s 2019-12-11T16:46:06.418 INFO:teuthology.orchestra.run.smithi079.stderr: 15300K .......... .......... .......... .......... .......... 20% 261M 0s 2019-12-11T16:46:06.418 INFO:teuthology.orchestra.run.smithi079.stderr: 15350K .......... .......... .......... .......... .......... 20% 174M 0s 2019-12-11T16:46:06.418 INFO:teuthology.orchestra.run.smithi079.stderr: 15400K .......... .......... .......... .......... .......... 21% 233M 0s 2019-12-11T16:46:06.418 INFO:teuthology.orchestra.run.smithi079.stderr: 15450K .......... .......... .......... .......... .......... 21% 252M 0s 2019-12-11T16:46:06.418 INFO:teuthology.orchestra.run.smithi079.stderr: 15500K .......... .......... .......... .......... .......... 21% 223M 0s 2019-12-11T16:46:06.419 INFO:teuthology.orchestra.run.smithi079.stderr: 15550K .......... .......... .......... .......... .......... 21% 180M 0s 2019-12-11T16:46:06.419 INFO:teuthology.orchestra.run.smithi079.stderr: 15600K .......... .......... .......... .......... .......... 21% 218M 0s 2019-12-11T16:46:06.419 INFO:teuthology.orchestra.run.smithi079.stderr: 15650K .......... .......... .......... .......... .......... 21% 272M 0s 2019-12-11T16:46:06.419 INFO:teuthology.orchestra.run.smithi079.stderr: 15700K .......... .......... .......... .......... .......... 21% 229M 0s 2019-12-11T16:46:06.419 INFO:teuthology.orchestra.run.smithi079.stderr: 15750K .......... .......... .......... .......... .......... 21% 194M 0s 2019-12-11T16:46:06.420 INFO:teuthology.orchestra.run.smithi079.stderr: 15800K .......... .......... .......... .......... .......... 21% 212M 0s 2019-12-11T16:46:06.420 INFO:teuthology.orchestra.run.smithi079.stderr: 15850K .......... .......... .......... .......... .......... 21% 253M 0s 2019-12-11T16:46:06.420 INFO:teuthology.orchestra.run.smithi079.stderr: 15900K .......... .......... .......... .......... .......... 21% 327M 0s 2019-12-11T16:46:06.420 INFO:teuthology.orchestra.run.smithi079.stderr: 15950K .......... .......... .......... .......... .......... 21% 279M 0s 2019-12-11T16:46:06.420 INFO:teuthology.orchestra.run.smithi079.stderr: 16000K .......... .......... .......... .......... .......... 21% 309M 0s 2019-12-11T16:46:06.420 INFO:teuthology.orchestra.run.smithi079.stderr: 16050K .......... .......... .......... .......... .......... 21% 206M 0s 2019-12-11T16:46:06.420 INFO:teuthology.orchestra.run.smithi079.stderr: 16100K .......... .......... .......... .......... .......... 21% 192M 0s 2019-12-11T16:46:06.421 INFO:teuthology.orchestra.run.smithi079.stderr: 16150K .......... .......... .......... .......... .......... 22% 174M 0s 2019-12-11T16:46:06.421 INFO:teuthology.orchestra.run.smithi079.stderr: 16200K .......... .......... .......... .......... .......... 22% 193M 0s 2019-12-11T16:46:06.421 INFO:teuthology.orchestra.run.smithi079.stderr: 16250K .......... .......... .......... .......... .......... 22% 230M 0s 2019-12-11T16:46:06.421 INFO:teuthology.orchestra.run.smithi079.stderr: 16300K .......... .......... .......... .......... .......... 22% 142M 0s 2019-12-11T16:46:06.421 INFO:teuthology.orchestra.run.smithi079.stderr: 16350K .......... .......... .......... .......... .......... 22% 158M 0s 2019-12-11T16:46:06.421 INFO:teuthology.orchestra.run.smithi079.stderr: 16400K .......... .......... .......... .......... .......... 22% 159M 0s 2019-12-11T16:46:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 16450K .......... .......... .......... .......... .......... 22% 235M 0s 2019-12-11T16:46:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 16500K .......... .......... .......... .......... .......... 22% 227M 0s 2019-12-11T16:46:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 16550K .......... .......... .......... .......... .......... 22% 131M 0s 2019-12-11T16:46:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 16600K .......... .......... .......... .......... .......... 22% 138M 0s 2019-12-11T16:46:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 16650K .......... .......... .......... .......... .......... 22% 133M 0s 2019-12-11T16:46:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 16700K .......... .......... .......... .......... .......... 22% 211M 0s 2019-12-11T16:46:06.423 INFO:teuthology.orchestra.run.smithi079.stderr: 16750K .......... .......... .......... .......... .......... 22% 151M 0s 2019-12-11T16:46:06.423 INFO:teuthology.orchestra.run.smithi079.stderr: 16800K .......... .......... .......... .......... .......... 22% 225M 0s 2019-12-11T16:46:06.423 INFO:teuthology.orchestra.run.smithi079.stderr: 16850K .......... .......... .......... .......... .......... 22% 172M 0s 2019-12-11T16:46:06.423 INFO:teuthology.orchestra.run.smithi079.stderr: 16900K .......... .......... .......... .......... .......... 23% 162M 0s 2019-12-11T16:46:06.423 INFO:teuthology.orchestra.run.smithi079.stderr: 16950K .......... .......... .......... .......... .......... 23% 136M 0s 2019-12-11T16:46:06.423 INFO:teuthology.orchestra.run.smithi079.stderr: 17000K .......... .......... .......... .......... .......... 23% 196M 0s 2019-12-11T16:46:06.423 INFO:teuthology.orchestra.run.smithi079.stderr: 17050K .......... .......... .......... .......... .......... 23% 150M 0s 2019-12-11T16:46:06.424 INFO:teuthology.orchestra.run.smithi079.stderr: 17100K .......... .......... .......... .......... .......... 23% 185M 0s 2019-12-11T16:46:06.424 INFO:teuthology.orchestra.run.smithi079.stderr: 17150K .......... .......... .......... .......... .......... 23% 117M 0s 2019-12-11T16:46:06.424 INFO:teuthology.orchestra.run.smithi079.stderr: 17200K .......... .......... .......... .......... .......... 23% 163M 0s 2019-12-11T16:46:06.424 INFO:teuthology.orchestra.run.smithi079.stderr: 17250K .......... .......... .......... .......... .......... 23% 170M 0s 2019-12-11T16:46:06.424 INFO:teuthology.orchestra.run.smithi079.stderr: 17300K .......... .......... .......... .......... .......... 23% 236M 0s 2019-12-11T16:46:06.424 INFO:teuthology.orchestra.run.smithi079.stderr: 17350K .......... .......... .......... .......... .......... 23% 279M 0s 2019-12-11T16:46:06.424 INFO:teuthology.orchestra.run.smithi079.stderr: 17400K .......... .......... .......... .......... .......... 23% 268M 0s 2019-12-11T16:46:06.425 INFO:teuthology.orchestra.run.smithi079.stderr: 17450K .......... .......... .......... .......... .......... 23% 314M 0s 2019-12-11T16:46:06.425 INFO:teuthology.orchestra.run.smithi079.stderr: 17500K .......... .......... .......... .......... .......... 23% 295M 0s 2019-12-11T16:46:06.425 INFO:teuthology.orchestra.run.smithi079.stderr: 17550K .......... .......... .......... .......... .......... 23% 188M 0s 2019-12-11T16:46:06.425 INFO:teuthology.orchestra.run.smithi079.stderr: 17600K .......... .......... .......... .......... .......... 23% 200M 0s 2019-12-11T16:46:06.425 INFO:teuthology.orchestra.run.smithi079.stderr: 17650K .......... .......... .......... .......... .......... 24% 213M 0s 2019-12-11T16:46:06.425 INFO:teuthology.orchestra.run.smithi079.stderr: 17700K .......... .......... .......... .......... .......... 24% 238M 0s 2019-12-11T16:46:06.425 INFO:teuthology.orchestra.run.smithi079.stderr: 17750K .......... .......... .......... .......... .......... 24% 193M 0s 2019-12-11T16:46:06.426 INFO:teuthology.orchestra.run.smithi079.stderr: 17800K .......... .......... .......... .......... .......... 24% 242M 0s 2019-12-11T16:46:06.426 INFO:teuthology.orchestra.run.smithi079.stderr: 17850K .......... .......... .......... .......... .......... 24% 203M 0s 2019-12-11T16:46:06.426 INFO:teuthology.orchestra.run.smithi079.stderr: 17900K .......... .......... .......... .......... .......... 24% 265M 0s 2019-12-11T16:46:06.426 INFO:teuthology.orchestra.run.smithi079.stderr: 17950K .......... .......... .......... .......... .......... 24% 216M 0s 2019-12-11T16:46:06.426 INFO:teuthology.orchestra.run.smithi079.stderr: 18000K .......... .......... .......... .......... .......... 24% 207M 0s 2019-12-11T16:46:06.426 INFO:teuthology.orchestra.run.smithi079.stderr: 18050K .......... .......... .......... .......... .......... 24% 301M 0s 2019-12-11T16:46:06.426 INFO:teuthology.orchestra.run.smithi079.stderr: 18100K .......... .......... .......... .......... .......... 24% 364M 0s 2019-12-11T16:46:06.427 INFO:teuthology.orchestra.run.smithi079.stderr: 18150K .......... .......... .......... .......... .......... 24% 191M 0s 2019-12-11T16:46:06.427 INFO:teuthology.orchestra.run.smithi079.stderr: 18200K .......... .......... .......... .......... .......... 24% 206M 0s 2019-12-11T16:46:06.427 INFO:teuthology.orchestra.run.smithi079.stderr: 18250K .......... .......... .......... .......... .......... 24% 269M 0s 2019-12-11T16:46:06.427 INFO:teuthology.orchestra.run.smithi079.stderr: 18300K .......... .......... .......... .......... .......... 24% 256M 0s 2019-12-11T16:46:06.427 INFO:teuthology.orchestra.run.smithi079.stderr: 18350K .......... .......... .......... .......... .......... 25% 169M 0s 2019-12-11T16:46:06.427 INFO:teuthology.orchestra.run.smithi079.stderr: 18400K .......... .......... .......... .......... .......... 25% 267M 0s 2019-12-11T16:46:06.427 INFO:teuthology.orchestra.run.smithi079.stderr: 18450K .......... .......... .......... .......... .......... 25% 209M 0s 2019-12-11T16:46:06.428 INFO:teuthology.orchestra.run.smithi079.stderr: 18500K .......... .......... .......... .......... .......... 25% 212M 0s 2019-12-11T16:46:06.428 INFO:teuthology.orchestra.run.smithi079.stderr: 18550K .......... .......... .......... .......... .......... 25% 191M 0s 2019-12-11T16:46:06.428 INFO:teuthology.orchestra.run.smithi079.stderr: 18600K .......... .......... .......... .......... .......... 25% 229M 0s 2019-12-11T16:46:06.428 INFO:teuthology.orchestra.run.smithi079.stderr: 18650K .......... .......... .......... .......... .......... 25% 264M 0s 2019-12-11T16:46:06.428 INFO:teuthology.orchestra.run.smithi079.stderr: 18700K .......... .......... .......... .......... .......... 25% 233M 0s 2019-12-11T16:46:06.429 INFO:teuthology.orchestra.run.smithi079.stderr: 18750K .......... .......... .......... .......... .......... 25% 204M 0s 2019-12-11T16:46:06.429 INFO:teuthology.orchestra.run.smithi079.stderr: 18800K .......... .......... .......... .......... .......... 25% 223M 0s 2019-12-11T16:46:06.429 INFO:teuthology.orchestra.run.smithi079.stderr: 18850K .......... .......... .......... .......... .......... 25% 236M 0s 2019-12-11T16:46:06.429 INFO:teuthology.orchestra.run.smithi079.stderr: 18900K .......... .......... .......... .......... .......... 25% 206M 0s 2019-12-11T16:46:06.429 INFO:teuthology.orchestra.run.smithi079.stderr: 18950K .......... .......... .......... .......... .......... 25% 223M 0s 2019-12-11T16:46:06.429 INFO:teuthology.orchestra.run.smithi079.stderr: 19000K .......... .......... .......... .......... .......... 25% 274M 0s 2019-12-11T16:46:06.429 INFO:teuthology.orchestra.run.smithi079.stderr: 19050K .......... .......... .......... .......... .......... 25% 218M 0s 2019-12-11T16:46:06.430 INFO:teuthology.orchestra.run.smithi079.stderr: 19100K .......... .......... .......... .......... .......... 26% 277M 0s 2019-12-11T16:46:06.430 INFO:teuthology.orchestra.run.smithi079.stderr: 19150K .......... .......... .......... .......... .......... 26% 164M 0s 2019-12-11T16:46:06.430 INFO:teuthology.orchestra.run.smithi079.stderr: 19200K .......... .......... .......... .......... .......... 26% 225M 0s 2019-12-11T16:46:06.430 INFO:teuthology.orchestra.run.smithi079.stderr: 19250K .......... .......... .......... .......... .......... 26% 207M 0s 2019-12-11T16:46:06.430 INFO:teuthology.orchestra.run.smithi079.stderr: 19300K .......... .......... .......... .......... .......... 26% 222M 0s 2019-12-11T16:46:06.430 INFO:teuthology.orchestra.run.smithi079.stderr: 19350K .......... .......... .......... .......... .......... 26% 180M 0s 2019-12-11T16:46:06.430 INFO:teuthology.orchestra.run.smithi079.stderr: 19400K .......... .......... .......... .......... .......... 26% 289M 0s 2019-12-11T16:46:06.431 INFO:teuthology.orchestra.run.smithi079.stderr: 19450K .......... .......... .......... .......... .......... 26% 285M 0s 2019-12-11T16:46:06.431 INFO:teuthology.orchestra.run.smithi079.stderr: 19500K .......... .......... .......... .......... .......... 26% 193M 0s 2019-12-11T16:46:06.431 INFO:teuthology.orchestra.run.smithi079.stderr: 19550K .......... .......... .......... .......... .......... 26% 180M 0s 2019-12-11T16:46:06.431 INFO:teuthology.orchestra.run.smithi079.stderr: 19600K .......... .......... .......... .......... .......... 26% 256M 0s 2019-12-11T16:46:06.431 INFO:teuthology.orchestra.run.smithi079.stderr: 19650K .......... .......... .......... .......... .......... 26% 264M 0s 2019-12-11T16:46:06.431 INFO:teuthology.orchestra.run.smithi079.stderr: 19700K .......... .......... .......... .......... .......... 26% 225M 0s 2019-12-11T16:46:06.431 INFO:teuthology.orchestra.run.smithi079.stderr: 19750K .......... .......... .......... .......... .......... 26% 190M 0s 2019-12-11T16:46:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 19800K .......... .......... .......... .......... .......... 26% 265M 0s 2019-12-11T16:46:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 19850K .......... .......... .......... .......... .......... 27% 241M 0s 2019-12-11T16:46:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 19900K .......... .......... .......... .......... .......... 27% 198M 0s 2019-12-11T16:46:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 19950K .......... .......... .......... .......... .......... 27% 178M 0s 2019-12-11T16:46:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 20000K .......... .......... .......... .......... .......... 27% 224M 0s 2019-12-11T16:46:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 20050K .......... .......... .......... .......... .......... 27% 270M 0s 2019-12-11T16:46:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 20100K .......... .......... .......... .......... .......... 27% 284M 0s 2019-12-11T16:46:06.433 INFO:teuthology.orchestra.run.smithi079.stderr: 20150K .......... .......... .......... .......... .......... 27% 178M 0s 2019-12-11T16:46:06.433 INFO:teuthology.orchestra.run.smithi079.stderr: 20200K .......... .......... .......... .......... .......... 27% 244M 0s 2019-12-11T16:46:06.433 INFO:teuthology.orchestra.run.smithi079.stderr: 20250K .......... .......... .......... .......... .......... 27% 246M 0s 2019-12-11T16:46:06.433 INFO:teuthology.orchestra.run.smithi079.stderr: 20300K .......... .......... .......... .......... .......... 27% 225M 0s 2019-12-11T16:46:06.433 INFO:teuthology.orchestra.run.smithi079.stderr: 20350K .......... .......... .......... .......... .......... 27% 222M 0s 2019-12-11T16:46:06.433 INFO:teuthology.orchestra.run.smithi079.stderr: 20400K .......... .......... .......... .......... .......... 27% 200M 0s 2019-12-11T16:46:06.433 INFO:teuthology.orchestra.run.smithi079.stderr: 20450K .......... .......... .......... .......... .......... 27% 243M 0s 2019-12-11T16:46:06.434 INFO:teuthology.orchestra.run.smithi079.stderr: 20500K .......... .......... .......... .......... .......... 27% 264M 0s 2019-12-11T16:46:06.434 INFO:teuthology.orchestra.run.smithi079.stderr: 20550K .......... .......... .......... .......... .......... 28% 189M 0s 2019-12-11T16:46:06.434 INFO:teuthology.orchestra.run.smithi079.stderr: 20600K .......... .......... .......... .......... .......... 28% 258M 0s 2019-12-11T16:46:06.434 INFO:teuthology.orchestra.run.smithi079.stderr: 20650K .......... .......... .......... .......... .......... 28% 200M 0s 2019-12-11T16:46:06.434 INFO:teuthology.orchestra.run.smithi079.stderr: 20700K .......... .......... .......... .......... .......... 28% 276M 0s 2019-12-11T16:46:06.434 INFO:teuthology.orchestra.run.smithi079.stderr: 20750K .......... .......... .......... .......... .......... 28% 158M 0s 2019-12-11T16:46:06.434 INFO:teuthology.orchestra.run.smithi079.stderr: 20800K .......... .......... .......... .......... .......... 28% 263M 0s 2019-12-11T16:46:06.435 INFO:teuthology.orchestra.run.smithi079.stderr: 20850K .......... .......... .......... .......... .......... 28% 179M 0s 2019-12-11T16:46:06.435 INFO:teuthology.orchestra.run.smithi079.stderr: 20900K .......... .......... .......... .......... .......... 28% 285M 0s 2019-12-11T16:46:06.435 INFO:teuthology.orchestra.run.smithi079.stderr: 20950K .......... .......... .......... .......... .......... 28% 198M 0s 2019-12-11T16:46:06.435 INFO:teuthology.orchestra.run.smithi079.stderr: 21000K .......... .......... .......... .......... .......... 28% 251M 0s 2019-12-11T16:46:06.435 INFO:teuthology.orchestra.run.smithi079.stderr: 21050K .......... .......... .......... .......... .......... 28% 232M 0s 2019-12-11T16:46:06.435 INFO:teuthology.orchestra.run.smithi079.stderr: 21100K .......... .......... .......... .......... .......... 28% 223M 0s 2019-12-11T16:46:06.435 INFO:teuthology.orchestra.run.smithi079.stderr: 21150K .......... .......... .......... .......... .......... 28% 213M 0s 2019-12-11T16:46:06.436 INFO:teuthology.orchestra.run.smithi079.stderr: 21200K .......... .......... .......... .......... .......... 28% 220M 0s 2019-12-11T16:46:06.436 INFO:teuthology.orchestra.run.smithi079.stderr: 21250K .......... .......... .......... .......... .......... 28% 223M 0s 2019-12-11T16:46:06.436 INFO:teuthology.orchestra.run.smithi079.stderr: 21300K .......... .......... .......... .......... .......... 29% 276M 0s 2019-12-11T16:46:06.436 INFO:teuthology.orchestra.run.smithi079.stderr: 21350K .......... .......... .......... .......... .......... 29% 263M 0s 2019-12-11T16:46:06.436 INFO:teuthology.orchestra.run.smithi079.stderr: 21400K .......... .......... .......... .......... .......... 29% 210M 0s 2019-12-11T16:46:06.436 INFO:teuthology.orchestra.run.smithi079.stderr: 21450K .......... .......... .......... .......... .......... 29% 191M 0s 2019-12-11T16:46:06.437 INFO:teuthology.orchestra.run.smithi079.stderr: 21500K .......... .......... .......... .......... .......... 29% 383M 0s 2019-12-11T16:46:06.437 INFO:teuthology.orchestra.run.smithi079.stderr: 21550K .......... .......... .......... .......... .......... 29% 330M 0s 2019-12-11T16:46:06.437 INFO:teuthology.orchestra.run.smithi079.stderr: 21600K .......... .......... .......... .......... .......... 29% 311M 0s 2019-12-11T16:46:06.437 INFO:teuthology.orchestra.run.smithi079.stderr: 21650K .......... .......... .......... .......... .......... 29% 321M 0s 2019-12-11T16:46:06.437 INFO:teuthology.orchestra.run.smithi079.stderr: 21700K .......... .......... .......... .......... .......... 29% 300M 0s 2019-12-11T16:46:06.437 INFO:teuthology.orchestra.run.smithi079.stderr: 21750K .......... .......... .......... .......... .......... 29% 211M 0s 2019-12-11T16:46:06.437 INFO:teuthology.orchestra.run.smithi079.stderr: 21800K .......... .......... .......... .......... .......... 29% 343M 0s 2019-12-11T16:46:06.438 INFO:teuthology.orchestra.run.smithi079.stderr: 21850K .......... .......... .......... .......... .......... 29% 358M 0s 2019-12-11T16:46:06.438 INFO:teuthology.orchestra.run.smithi079.stderr: 21900K .......... .......... .......... .......... .......... 29% 270M 0s 2019-12-11T16:46:06.438 INFO:teuthology.orchestra.run.smithi079.stderr: 21950K .......... .......... .......... .......... .......... 29% 268M 0s 2019-12-11T16:46:06.438 INFO:teuthology.orchestra.run.smithi079.stderr: 22000K .......... .......... .......... .......... .......... 29% 330M 0s 2019-12-11T16:46:06.438 INFO:teuthology.orchestra.run.smithi079.stderr: 22050K .......... .......... .......... .......... .......... 30% 260M 0s 2019-12-11T16:46:06.438 INFO:teuthology.orchestra.run.smithi079.stderr: 22100K .......... .......... .......... .......... .......... 30% 229M 0s 2019-12-11T16:46:06.438 INFO:teuthology.orchestra.run.smithi079.stderr: 22150K .......... .......... .......... .......... .......... 30% 286M 0s 2019-12-11T16:46:06.439 INFO:teuthology.orchestra.run.smithi079.stderr: 22200K .......... .......... .......... .......... .......... 30% 347M 0s 2019-12-11T16:46:06.439 INFO:teuthology.orchestra.run.smithi079.stderr: 22250K .......... .......... .......... .......... .......... 30% 394M 0s 2019-12-11T16:46:06.439 INFO:teuthology.orchestra.run.smithi079.stderr: 22300K .......... .......... .......... .......... .......... 30% 388M 0s 2019-12-11T16:46:06.439 INFO:teuthology.orchestra.run.smithi079.stderr: 22350K .......... .......... .......... .......... .......... 30% 232M 0s 2019-12-11T16:46:06.439 INFO:teuthology.orchestra.run.smithi079.stderr: 22400K .......... .......... .......... .......... .......... 30% 364M 0s 2019-12-11T16:46:06.439 INFO:teuthology.orchestra.run.smithi079.stderr: 22450K .......... .......... .......... .......... .......... 30% 350M 0s 2019-12-11T16:46:06.440 INFO:teuthology.orchestra.run.smithi079.stderr: 22500K .......... .......... .......... .......... .......... 30% 200M 0s 2019-12-11T16:46:06.440 INFO:teuthology.orchestra.run.smithi079.stderr: 22550K .......... .......... .......... .......... .......... 30% 190M 0s 2019-12-11T16:46:06.440 INFO:teuthology.orchestra.run.smithi079.stderr: 22600K .......... .......... .......... .......... .......... 30% 223M 0s 2019-12-11T16:46:06.440 INFO:teuthology.orchestra.run.smithi079.stderr: 22650K .......... .......... .......... .......... .......... 30% 269M 0s 2019-12-11T16:46:06.440 INFO:teuthology.orchestra.run.smithi079.stderr: 22700K .......... .......... .......... .......... .......... 30% 227M 0s 2019-12-11T16:46:06.440 INFO:teuthology.orchestra.run.smithi079.stderr: 22750K .......... .......... .......... .......... .......... 30% 188M 0s 2019-12-11T16:46:06.441 INFO:teuthology.orchestra.run.smithi079.stderr: 22800K .......... .......... .......... .......... .......... 31% 221M 0s 2019-12-11T16:46:06.441 INFO:teuthology.orchestra.run.smithi079.stderr: 22850K .......... .......... .......... .......... .......... 31% 271M 0s 2019-12-11T16:46:06.441 INFO:teuthology.orchestra.run.smithi079.stderr: 22900K .......... .......... .......... .......... .......... 31% 277M 0s 2019-12-11T16:46:06.441 INFO:teuthology.orchestra.run.smithi079.stderr: 22950K .......... .......... .......... .......... .......... 31% 190M 0s 2019-12-11T16:46:06.441 INFO:teuthology.orchestra.run.smithi079.stderr: 23000K .......... .......... .......... .......... .......... 31% 226M 0s 2019-12-11T16:46:06.441 INFO:teuthology.orchestra.run.smithi079.stderr: 23050K .......... .......... .......... .......... .......... 31% 240M 0s 2019-12-11T16:46:06.441 INFO:teuthology.orchestra.run.smithi079.stderr: 23100K .......... .......... .......... .......... .......... 31% 274M 0s 2019-12-11T16:46:06.442 INFO:teuthology.orchestra.run.smithi079.stderr: 23150K .......... .......... .......... .......... .......... 31% 208M 0s 2019-12-11T16:46:06.442 INFO:teuthology.orchestra.run.smithi079.stderr: 23200K .......... .......... .......... .......... .......... 31% 212M 0s 2019-12-11T16:46:06.442 INFO:teuthology.orchestra.run.smithi079.stderr: 23250K .......... .......... .......... .......... .......... 31% 235M 0s 2019-12-11T16:46:06.442 INFO:teuthology.orchestra.run.smithi079.stderr: 23300K .......... .......... .......... .......... .......... 31% 231M 0s 2019-12-11T16:46:06.442 INFO:teuthology.orchestra.run.smithi079.stderr: 23350K .......... .......... .......... .......... .......... 31% 224M 0s 2019-12-11T16:46:06.442 INFO:teuthology.orchestra.run.smithi079.stderr: 23400K .......... .......... .......... .......... .......... 31% 249M 0s 2019-12-11T16:46:06.442 INFO:teuthology.orchestra.run.smithi079.stderr: 23450K .......... .......... .......... .......... .......... 31% 214M 0s 2019-12-11T16:46:06.443 INFO:teuthology.orchestra.run.smithi079.stderr: 23500K .......... .......... .......... .......... .......... 32% 273M 0s 2019-12-11T16:46:06.443 INFO:teuthology.orchestra.run.smithi079.stderr: 23550K .......... .......... .......... .......... .......... 32% 179M 0s 2019-12-11T16:46:06.443 INFO:teuthology.orchestra.run.smithi079.stderr: 23600K .......... .......... .......... .......... .......... 32% 233M 0s 2019-12-11T16:46:06.443 INFO:teuthology.orchestra.run.smithi079.stderr: 23650K .......... .......... .......... .......... .......... 32% 221M 0s 2019-12-11T16:46:06.443 INFO:teuthology.orchestra.run.smithi079.stderr: 23700K .......... .......... .......... .......... .......... 32% 198M 0s 2019-12-11T16:46:06.443 INFO:teuthology.orchestra.run.smithi079.stderr: 23750K .......... .......... .......... .......... .......... 32% 196M 0s 2019-12-11T16:46:06.443 INFO:teuthology.orchestra.run.smithi079.stderr: 23800K .......... .......... .......... .......... .......... 32% 205M 0s 2019-12-11T16:46:06.444 INFO:teuthology.orchestra.run.smithi079.stderr: 23850K .......... .......... .......... .......... .......... 32% 203M 0s 2019-12-11T16:46:06.444 INFO:teuthology.orchestra.run.smithi079.stderr: 23900K .......... .......... .......... .......... .......... 32% 203M 0s 2019-12-11T16:46:06.444 INFO:teuthology.orchestra.run.smithi079.stderr: 23950K .......... .......... .......... .......... .......... 32% 199M 0s 2019-12-11T16:46:06.444 INFO:teuthology.orchestra.run.smithi079.stderr: 24000K .......... .......... .......... .......... .......... 32% 255M 0s 2019-12-11T16:46:06.444 INFO:teuthology.orchestra.run.smithi079.stderr: 24050K .......... .......... .......... .......... .......... 32% 257M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24100K .......... .......... .......... .......... .......... 32% 188M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24150K .......... .......... .......... .......... .......... 32% 204M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24200K .......... .......... .......... .......... .......... 32% 225M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24250K .......... .......... .......... .......... .......... 33% 230M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24300K .......... .......... .......... .......... .......... 33% 228M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24350K .......... .......... .......... .......... .......... 33% 187M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24400K .......... .......... .......... .......... .......... 33% 209M 0s 2019-12-11T16:46:06.445 INFO:teuthology.orchestra.run.smithi079.stderr: 24450K .......... .......... .......... .......... .......... 33% 235M 0s 2019-12-11T16:46:06.446 INFO:teuthology.orchestra.run.smithi079.stderr: 24500K .......... .......... .......... .......... .......... 33% 216M 0s 2019-12-11T16:46:06.446 INFO:teuthology.orchestra.run.smithi079.stderr: 24550K .......... .......... .......... .......... .......... 33% 216M 0s 2019-12-11T16:46:06.446 INFO:teuthology.orchestra.run.smithi079.stderr: 24600K .......... .......... .......... .......... .......... 33% 196M 0s 2019-12-11T16:46:06.446 INFO:teuthology.orchestra.run.smithi079.stderr: 24650K .......... .......... .......... .......... .......... 33% 256M 0s 2019-12-11T16:46:06.446 INFO:teuthology.orchestra.run.smithi079.stderr: 24700K .......... .......... .......... .......... .......... 33% 215M 0s 2019-12-11T16:46:06.446 INFO:teuthology.orchestra.run.smithi079.stderr: 24750K .......... .......... .......... .......... .......... 33% 189M 0s 2019-12-11T16:46:06.446 INFO:teuthology.orchestra.run.smithi079.stderr: 24800K .......... .......... .......... .......... .......... 33% 240M 0s 2019-12-11T16:46:06.447 INFO:teuthology.orchestra.run.smithi079.stderr: 24850K .......... .......... .......... .......... .......... 33% 210M 0s 2019-12-11T16:46:06.447 INFO:teuthology.orchestra.run.smithi079.stderr: 24900K .......... .......... .......... .......... .......... 33% 234M 0s 2019-12-11T16:46:06.447 INFO:teuthology.orchestra.run.smithi079.stderr: 24950K .......... .......... .......... .......... .......... 33% 196M 0s 2019-12-11T16:46:06.447 INFO:teuthology.orchestra.run.smithi079.stderr: 25000K .......... .......... .......... .......... .......... 34% 246M 0s 2019-12-11T16:46:06.447 INFO:teuthology.orchestra.run.smithi079.stderr: 25050K .......... .......... .......... .......... .......... 34% 283M 0s 2019-12-11T16:46:06.447 INFO:teuthology.orchestra.run.smithi079.stderr: 25100K .......... .......... .......... .......... .......... 34% 214M 0s 2019-12-11T16:46:06.448 INFO:teuthology.orchestra.run.smithi079.stderr: 25150K .......... .......... .......... .......... .......... 34% 203M 0s 2019-12-11T16:46:06.448 INFO:teuthology.orchestra.run.smithi079.stderr: 25200K .......... .......... .......... .......... .......... 34% 221M 0s 2019-12-11T16:46:06.448 INFO:teuthology.orchestra.run.smithi079.stderr: 25250K .......... .......... .......... .......... .......... 34% 247M 0s 2019-12-11T16:46:06.448 INFO:teuthology.orchestra.run.smithi079.stderr: 25300K .......... .......... .......... .......... .......... 34% 253M 0s 2019-12-11T16:46:06.448 INFO:teuthology.orchestra.run.smithi079.stderr: 25350K .......... .......... .......... .......... .......... 34% 220M 0s 2019-12-11T16:46:06.448 INFO:teuthology.orchestra.run.smithi079.stderr: 25400K .......... .......... .......... .......... .......... 34% 241M 0s 2019-12-11T16:46:06.448 INFO:teuthology.orchestra.run.smithi079.stderr: 25450K .......... .......... .......... .......... .......... 34% 251M 0s 2019-12-11T16:46:06.449 INFO:teuthology.orchestra.run.smithi079.stderr: 25500K .......... .......... .......... .......... .......... 34% 248M 0s 2019-12-11T16:46:06.449 INFO:teuthology.orchestra.run.smithi079.stderr: 25550K .......... .......... .......... .......... .......... 34% 202M 0s 2019-12-11T16:46:06.449 INFO:teuthology.orchestra.run.smithi079.stderr: 25600K .......... .......... .......... .......... .......... 34% 244M 0s 2019-12-11T16:46:06.449 INFO:teuthology.orchestra.run.smithi079.stderr: 25650K .......... .......... .......... .......... .......... 34% 290M 0s 2019-12-11T16:46:06.449 INFO:teuthology.orchestra.run.smithi079.stderr: 25700K .......... .......... .......... .......... .......... 35% 233M 0s 2019-12-11T16:46:06.449 INFO:teuthology.orchestra.run.smithi079.stderr: 25750K .......... .......... .......... .......... .......... 35% 236M 0s 2019-12-11T16:46:06.450 INFO:teuthology.orchestra.run.smithi079.stderr: 25800K .......... .......... .......... .......... .......... 35% 212M 0s 2019-12-11T16:46:06.450 INFO:teuthology.orchestra.run.smithi079.stderr: 25850K .......... .......... .......... .......... .......... 35% 242M 0s 2019-12-11T16:46:06.450 INFO:teuthology.orchestra.run.smithi079.stderr: 25900K .......... .......... .......... .......... .......... 35% 263M 0s 2019-12-11T16:46:06.450 INFO:teuthology.orchestra.run.smithi079.stderr: 25950K .......... .......... .......... .......... .......... 35% 199M 0s 2019-12-11T16:46:06.450 INFO:teuthology.orchestra.run.smithi079.stderr: 26000K .......... .......... .......... .......... .......... 35% 257M 0s 2019-12-11T16:46:06.450 INFO:teuthology.orchestra.run.smithi079.stderr: 26050K .......... .......... .......... .......... .......... 35% 227M 0s 2019-12-11T16:46:06.450 INFO:teuthology.orchestra.run.smithi079.stderr: 26100K .......... .......... .......... .......... .......... 35% 272M 0s 2019-12-11T16:46:06.451 INFO:teuthology.orchestra.run.smithi079.stderr: 26150K .......... .......... .......... .......... .......... 35% 222M 0s 2019-12-11T16:46:06.451 INFO:teuthology.orchestra.run.smithi079.stderr: 26200K .......... .......... .......... .......... .......... 35% 194M 0s 2019-12-11T16:46:06.451 INFO:teuthology.orchestra.run.smithi079.stderr: 26250K .......... .......... .......... .......... .......... 35% 219M 0s 2019-12-11T16:46:06.451 INFO:teuthology.orchestra.run.smithi079.stderr: 26300K .......... .......... .......... .......... .......... 35% 216M 0s 2019-12-11T16:46:06.451 INFO:teuthology.orchestra.run.smithi079.stderr: 26350K .......... .......... .......... .......... .......... 35% 168M 0s 2019-12-11T16:46:06.451 INFO:teuthology.orchestra.run.smithi079.stderr: 26400K .......... .......... .......... .......... .......... 35% 240M 0s 2019-12-11T16:46:06.451 INFO:teuthology.orchestra.run.smithi079.stderr: 26450K .......... .......... .......... .......... .......... 36% 203M 0s 2019-12-11T16:46:06.452 INFO:teuthology.orchestra.run.smithi079.stderr: 26500K .......... .......... .......... .......... .......... 36% 280M 0s 2019-12-11T16:46:06.452 INFO:teuthology.orchestra.run.smithi079.stderr: 26550K .......... .......... .......... .......... .......... 36% 202M 0s 2019-12-11T16:46:06.452 INFO:teuthology.orchestra.run.smithi079.stderr: 26600K .......... .......... .......... .......... .......... 36% 227M 0s 2019-12-11T16:46:06.452 INFO:teuthology.orchestra.run.smithi079.stderr: 26650K .......... .......... .......... .......... .......... 36% 240M 0s 2019-12-11T16:46:06.452 INFO:teuthology.orchestra.run.smithi079.stderr: 26700K .......... .......... .......... .......... .......... 36% 259M 0s 2019-12-11T16:46:06.452 INFO:teuthology.orchestra.run.smithi079.stderr: 26750K .......... .......... .......... .......... .......... 36% 198M 0s 2019-12-11T16:46:06.452 INFO:teuthology.orchestra.run.smithi079.stderr: 26800K .......... .......... .......... .......... .......... 36% 246M 0s 2019-12-11T16:46:06.453 INFO:teuthology.orchestra.run.smithi079.stderr: 26850K .......... .......... .......... .......... .......... 36% 207M 0s 2019-12-11T16:46:06.453 INFO:teuthology.orchestra.run.smithi079.stderr: 26900K .......... .......... .......... .......... .......... 36% 231M 0s 2019-12-11T16:46:06.453 INFO:teuthology.orchestra.run.smithi079.stderr: 26950K .......... .......... .......... .......... .......... 36% 232M 0s 2019-12-11T16:46:06.453 INFO:teuthology.orchestra.run.smithi079.stderr: 27000K .......... .......... .......... .......... .......... 36% 219M 0s 2019-12-11T16:46:06.453 INFO:teuthology.orchestra.run.smithi079.stderr: 27050K .......... .......... .......... .......... .......... 36% 234M 0s 2019-12-11T16:46:06.453 INFO:teuthology.orchestra.run.smithi079.stderr: 27100K .......... .......... .......... .......... .......... 36% 207M 0s 2019-12-11T16:46:06.453 INFO:teuthology.orchestra.run.smithi079.stderr: 27150K .......... .......... .......... .......... .......... 36% 186M 0s 2019-12-11T16:46:06.454 INFO:teuthology.orchestra.run.smithi079.stderr: 27200K .......... .......... .......... .......... .......... 37% 271M 0s 2019-12-11T16:46:06.454 INFO:teuthology.orchestra.run.smithi079.stderr: 27250K .......... .......... .......... .......... .......... 37% 190M 0s 2019-12-11T16:46:06.454 INFO:teuthology.orchestra.run.smithi079.stderr: 27300K .......... .......... .......... .......... .......... 37% 238M 0s 2019-12-11T16:46:06.454 INFO:teuthology.orchestra.run.smithi079.stderr: 27350K .......... .......... .......... .......... .......... 37% 247M 0s 2019-12-11T16:46:06.454 INFO:teuthology.orchestra.run.smithi079.stderr: 27400K .......... .......... .......... .......... .......... 37% 266M 0s 2019-12-11T16:46:06.454 INFO:teuthology.orchestra.run.smithi079.stderr: 27450K .......... .......... .......... .......... .......... 37% 234M 0s 2019-12-11T16:46:06.454 INFO:teuthology.orchestra.run.smithi079.stderr: 27500K .......... .......... .......... .......... .......... 37% 200M 0s 2019-12-11T16:46:06.455 INFO:teuthology.orchestra.run.smithi079.stderr: 27550K .......... .......... .......... .......... .......... 37% 199M 0s 2019-12-11T16:46:06.455 INFO:teuthology.orchestra.run.smithi079.stderr: 27600K .......... .......... .......... .......... .......... 37% 233M 0s 2019-12-11T16:46:06.455 INFO:teuthology.orchestra.run.smithi079.stderr: 27650K .......... .......... .......... .......... .......... 37% 264M 0s 2019-12-11T16:46:06.455 INFO:teuthology.orchestra.run.smithi079.stderr: 27700K .......... .......... .......... .......... .......... 37% 196M 0s 2019-12-11T16:46:06.455 INFO:teuthology.orchestra.run.smithi079.stderr: 27750K .......... .......... .......... .......... .......... 37% 204M 0s 2019-12-11T16:46:06.455 INFO:teuthology.orchestra.run.smithi079.stderr: 27800K .......... .......... .......... .......... .......... 37% 248M 0s 2019-12-11T16:46:06.456 INFO:teuthology.orchestra.run.smithi079.stderr: 27850K .......... .......... .......... .......... .......... 37% 239M 0s 2019-12-11T16:46:06.456 INFO:teuthology.orchestra.run.smithi079.stderr: 27900K .......... .......... .......... .......... .......... 37% 242M 0s 2019-12-11T16:46:06.456 INFO:teuthology.orchestra.run.smithi079.stderr: 27950K .......... .......... .......... .......... .......... 38% 186M 0s 2019-12-11T16:46:06.456 INFO:teuthology.orchestra.run.smithi079.stderr: 28000K .......... .......... .......... .......... .......... 38% 249M 0s 2019-12-11T16:46:06.456 INFO:teuthology.orchestra.run.smithi079.stderr: 28050K .......... .......... .......... .......... .......... 38% 276M 0s 2019-12-11T16:46:06.456 INFO:teuthology.orchestra.run.smithi079.stderr: 28100K .......... .......... .......... .......... .......... 38% 247M 0s 2019-12-11T16:46:06.456 INFO:teuthology.orchestra.run.smithi079.stderr: 28150K .......... .......... .......... .......... .......... 38% 217M 0s 2019-12-11T16:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr: 28200K .......... .......... .......... .......... .......... 38% 277M 0s 2019-12-11T16:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr: 28250K .......... .......... .......... .......... .......... 38% 251M 0s 2019-12-11T16:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr: 28300K .......... .......... .......... .......... .......... 38% 200M 0s 2019-12-11T16:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr: 28350K .......... .......... .......... .......... .......... 38% 178M 0s 2019-12-11T16:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr: 28400K .......... .......... .......... .......... .......... 38% 233M 0s 2019-12-11T16:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr: 28450K .......... .......... .......... .......... .......... 38% 220M 0s 2019-12-11T16:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr: 28500K .......... .......... .......... .......... .......... 38% 280M 0s 2019-12-11T16:46:06.458 INFO:teuthology.orchestra.run.smithi079.stderr: 28550K .......... .......... .......... .......... .......... 38% 206M 0s 2019-12-11T16:46:06.458 INFO:teuthology.orchestra.run.smithi079.stderr: 28600K .......... .......... .......... .......... .......... 38% 232M 0s 2019-12-11T16:46:06.458 INFO:teuthology.orchestra.run.smithi079.stderr: 28650K .......... .......... .......... .......... .......... 39% 228M 0s 2019-12-11T16:46:06.458 INFO:teuthology.orchestra.run.smithi079.stderr: 28700K .......... .......... .......... .......... .......... 39% 239M 0s 2019-12-11T16:46:06.458 INFO:teuthology.orchestra.run.smithi079.stderr: 28750K .......... .......... .......... .......... .......... 39% 212M 0s 2019-12-11T16:46:06.458 INFO:teuthology.orchestra.run.smithi079.stderr: 28800K .......... .......... .......... .......... .......... 39% 242M 0s 2019-12-11T16:46:06.458 INFO:teuthology.orchestra.run.smithi079.stderr: 28850K .......... .......... .......... .......... .......... 39% 226M 0s 2019-12-11T16:46:06.459 INFO:teuthology.orchestra.run.smithi079.stderr: 28900K .......... .......... .......... .......... .......... 39% 223M 0s 2019-12-11T16:46:06.459 INFO:teuthology.orchestra.run.smithi079.stderr: 28950K .......... .......... .......... .......... .......... 39% 230M 0s 2019-12-11T16:46:06.459 INFO:teuthology.orchestra.run.smithi079.stderr: 29000K .......... .......... .......... .......... .......... 39% 244M 0s 2019-12-11T16:46:06.459 INFO:teuthology.orchestra.run.smithi079.stderr: 29050K .......... .......... .......... .......... .......... 39% 243M 0s 2019-12-11T16:46:06.459 INFO:teuthology.orchestra.run.smithi079.stderr: 29100K .......... .......... .......... .......... .......... 39% 280M 0s 2019-12-11T16:46:06.459 INFO:teuthology.orchestra.run.smithi079.stderr: 29150K .......... .......... .......... .......... .......... 39% 189M 0s 2019-12-11T16:46:06.459 INFO:teuthology.orchestra.run.smithi079.stderr: 29200K .......... .......... .......... .......... .......... 39% 217M 0s 2019-12-11T16:46:06.460 INFO:teuthology.orchestra.run.smithi079.stderr: 29250K .......... .......... .......... .......... .......... 39% 224M 0s 2019-12-11T16:46:06.460 INFO:teuthology.orchestra.run.smithi079.stderr: 29300K .......... .......... .......... .......... .......... 39% 214M 0s 2019-12-11T16:46:06.460 INFO:teuthology.orchestra.run.smithi079.stderr: 29350K .......... .......... .......... .......... .......... 39% 261M 0s 2019-12-11T16:46:06.460 INFO:teuthology.orchestra.run.smithi079.stderr: 29400K .......... .......... .......... .......... .......... 40% 269M 0s 2019-12-11T16:46:06.460 INFO:teuthology.orchestra.run.smithi079.stderr: 29450K .......... .......... .......... .......... .......... 40% 191M 0s 2019-12-11T16:46:06.460 INFO:teuthology.orchestra.run.smithi079.stderr: 29500K .......... .......... .......... .......... .......... 40% 210M 0s 2019-12-11T16:46:06.460 INFO:teuthology.orchestra.run.smithi079.stderr: 29550K .......... .......... .......... .......... .......... 40% 203M 0s 2019-12-11T16:46:06.461 INFO:teuthology.orchestra.run.smithi079.stderr: 29600K .......... .......... .......... .......... .......... 40% 247M 0s 2019-12-11T16:46:06.461 INFO:teuthology.orchestra.run.smithi079.stderr: 29650K .......... .......... .......... .......... .......... 40% 225M 0s 2019-12-11T16:46:06.461 INFO:teuthology.orchestra.run.smithi079.stderr: 29700K .......... .......... .......... .......... .......... 40% 195M 0s 2019-12-11T16:46:06.461 INFO:teuthology.orchestra.run.smithi079.stderr: 29750K .......... .......... .......... .......... .......... 40% 246M 0s 2019-12-11T16:46:06.461 INFO:teuthology.orchestra.run.smithi079.stderr: 29800K .......... .......... .......... .......... .......... 40% 262M 0s 2019-12-11T16:46:06.461 INFO:teuthology.orchestra.run.smithi079.stderr: 29850K .......... .......... .......... .......... .......... 40% 208M 0s 2019-12-11T16:46:06.461 INFO:teuthology.orchestra.run.smithi079.stderr: 29900K .......... .......... .......... .......... .......... 40% 272M 0s 2019-12-11T16:46:06.462 INFO:teuthology.orchestra.run.smithi079.stderr: 29950K .......... .......... .......... .......... .......... 40% 240M 0s 2019-12-11T16:46:06.462 INFO:teuthology.orchestra.run.smithi079.stderr: 30000K .......... .......... .......... .......... .......... 40% 201M 0s 2019-12-11T16:46:06.462 INFO:teuthology.orchestra.run.smithi079.stderr: 30050K .......... .......... .......... .......... .......... 40% 239M 0s 2019-12-11T16:46:06.462 INFO:teuthology.orchestra.run.smithi079.stderr: 30100K .......... .......... .......... .......... .......... 40% 266M 0s 2019-12-11T16:46:06.462 INFO:teuthology.orchestra.run.smithi079.stderr: 30150K .......... .......... .......... .......... .......... 41% 204M 0s 2019-12-11T16:46:06.462 INFO:teuthology.orchestra.run.smithi079.stderr: 30200K .......... .......... .......... .......... .......... 41% 217M 0s 2019-12-11T16:46:06.462 INFO:teuthology.orchestra.run.smithi079.stderr: 30250K .......... .......... .......... .......... .......... 41% 256M 0s 2019-12-11T16:46:06.463 INFO:teuthology.orchestra.run.smithi079.stderr: 30300K .......... .......... .......... .......... .......... 41% 194M 0s 2019-12-11T16:46:06.463 INFO:teuthology.orchestra.run.smithi079.stderr: 30350K .......... .......... .......... .......... .......... 41% 195M 0s 2019-12-11T16:46:06.463 INFO:teuthology.orchestra.run.smithi079.stderr: 30400K .......... .......... .......... .......... .......... 41% 235M 0s 2019-12-11T16:46:06.463 INFO:teuthology.orchestra.run.smithi079.stderr: 30450K .......... .......... .......... .......... .......... 41% 249M 0s 2019-12-11T16:46:06.463 INFO:teuthology.orchestra.run.smithi079.stderr: 30500K .......... .......... .......... .......... .......... 41% 255M 0s 2019-12-11T16:46:06.463 INFO:teuthology.orchestra.run.smithi079.stderr: 30550K .......... .......... .......... .......... .......... 41% 202M 0s 2019-12-11T16:46:06.463 INFO:teuthology.orchestra.run.smithi079.stderr: 30600K .......... .......... .......... .......... .......... 41% 217M 0s 2019-12-11T16:46:06.464 INFO:teuthology.orchestra.run.smithi079.stderr: 30650K .......... .......... .......... .......... .......... 41% 222M 0s 2019-12-11T16:46:06.464 INFO:teuthology.orchestra.run.smithi079.stderr: 30700K .......... .......... .......... .......... .......... 41% 208M 0s 2019-12-11T16:46:06.464 INFO:teuthology.orchestra.run.smithi079.stderr: 30750K .......... .......... .......... .......... .......... 41% 195M 0s 2019-12-11T16:46:06.464 INFO:teuthology.orchestra.run.smithi079.stderr: 30800K .......... .......... .......... .......... .......... 41% 251M 0s 2019-12-11T16:46:06.464 INFO:teuthology.orchestra.run.smithi079.stderr: 30850K .......... .......... .......... .......... .......... 42% 252M 0s 2019-12-11T16:46:06.464 INFO:teuthology.orchestra.run.smithi079.stderr: 30900K .......... .......... .......... .......... .......... 42% 253M 0s 2019-12-11T16:46:06.465 INFO:teuthology.orchestra.run.smithi079.stderr: 30950K .......... .......... .......... .......... .......... 42% 188M 0s 2019-12-11T16:46:06.465 INFO:teuthology.orchestra.run.smithi079.stderr: 31000K .......... .......... .......... .......... .......... 42% 215M 0s 2019-12-11T16:46:06.465 INFO:teuthology.orchestra.run.smithi079.stderr: 31050K .......... .......... .......... .......... .......... 42% 260M 0s 2019-12-11T16:46:06.465 INFO:teuthology.orchestra.run.smithi079.stderr: 31100K .......... .......... .......... .......... .......... 42% 278M 0s 2019-12-11T16:46:06.465 INFO:teuthology.orchestra.run.smithi079.stderr: 31150K .......... .......... .......... .......... .......... 42% 171M 0s 2019-12-11T16:46:06.465 INFO:teuthology.orchestra.run.smithi079.stderr: 31200K .......... .......... .......... .......... .......... 42% 207M 0s 2019-12-11T16:46:06.465 INFO:teuthology.orchestra.run.smithi079.stderr: 31250K .......... .......... .......... .......... .......... 42% 222M 0s 2019-12-11T16:46:06.466 INFO:teuthology.orchestra.run.smithi079.stderr: 31300K .......... .......... .......... .......... .......... 42% 280M 0s 2019-12-11T16:46:06.466 INFO:teuthology.orchestra.run.smithi079.stderr: 31350K .......... .......... .......... .......... .......... 42% 226M 0s 2019-12-11T16:46:06.466 INFO:teuthology.orchestra.run.smithi079.stderr: 31400K .......... .......... .......... .......... .......... 42% 221M 0s 2019-12-11T16:46:06.466 INFO:teuthology.orchestra.run.smithi079.stderr: 31450K .......... .......... .......... .......... .......... 42% 240M 0s 2019-12-11T16:46:06.466 INFO:teuthology.orchestra.run.smithi079.stderr: 31500K .......... .......... .......... .......... .......... 42% 273M 0s 2019-12-11T16:46:06.466 INFO:teuthology.orchestra.run.smithi079.stderr: 31550K .......... .......... .......... .......... .......... 42% 187M 0s 2019-12-11T16:46:06.467 INFO:teuthology.orchestra.run.smithi079.stderr: 31600K .......... .......... .......... .......... .......... 43% 227M 0s 2019-12-11T16:46:06.467 INFO:teuthology.orchestra.run.smithi079.stderr: 31650K .......... .......... .......... .......... .......... 43% 288M 0s 2019-12-11T16:46:06.467 INFO:teuthology.orchestra.run.smithi079.stderr: 31700K .......... .......... .......... .......... .......... 43% 223M 0s 2019-12-11T16:46:06.467 INFO:teuthology.orchestra.run.smithi079.stderr: 31750K .......... .......... .......... .......... .......... 43% 179M 0s 2019-12-11T16:46:06.467 INFO:teuthology.orchestra.run.smithi079.stderr: 31800K .......... .......... .......... .......... .......... 43% 173M 0s 2019-12-11T16:46:06.467 INFO:teuthology.orchestra.run.smithi079.stderr: 31850K .......... .......... .......... .......... .......... 43% 259M 0s 2019-12-11T16:46:06.468 INFO:teuthology.orchestra.run.smithi079.stderr: 31900K .......... .......... .......... .......... .......... 43% 220M 0s 2019-12-11T16:46:06.468 INFO:teuthology.orchestra.run.smithi079.stderr: 31950K .......... .......... .......... .......... .......... 43% 169M 0s 2019-12-11T16:46:06.468 INFO:teuthology.orchestra.run.smithi079.stderr: 32000K .......... .......... .......... .......... .......... 43% 232M 0s 2019-12-11T16:46:06.468 INFO:teuthology.orchestra.run.smithi079.stderr: 32050K .......... .......... .......... .......... .......... 43% 238M 0s 2019-12-11T16:46:06.468 INFO:teuthology.orchestra.run.smithi079.stderr: 32100K .......... .......... .......... .......... .......... 43% 199M 0s 2019-12-11T16:46:06.468 INFO:teuthology.orchestra.run.smithi079.stderr: 32150K .......... .......... .......... .......... .......... 43% 222M 0s 2019-12-11T16:46:06.468 INFO:teuthology.orchestra.run.smithi079.stderr: 32200K .......... .......... .......... .......... .......... 43% 246M 0s 2019-12-11T16:46:06.469 INFO:teuthology.orchestra.run.smithi079.stderr: 32250K .......... .......... .......... .......... .......... 43% 254M 0s 2019-12-11T16:46:06.469 INFO:teuthology.orchestra.run.smithi079.stderr: 32300K .......... .......... .......... .......... .......... 43% 215M 0s 2019-12-11T16:46:06.469 INFO:teuthology.orchestra.run.smithi079.stderr: 32350K .......... .......... .......... .......... .......... 44% 188M 0s 2019-12-11T16:46:06.469 INFO:teuthology.orchestra.run.smithi079.stderr: 32400K .......... .......... .......... .......... .......... 44% 250M 0s 2019-12-11T16:46:06.469 INFO:teuthology.orchestra.run.smithi079.stderr: 32450K .......... .......... .......... .......... .......... 44% 248M 0s 2019-12-11T16:46:06.469 INFO:teuthology.orchestra.run.smithi079.stderr: 32500K .......... .......... .......... .......... .......... 44% 205M 0s 2019-12-11T16:46:06.469 INFO:teuthology.orchestra.run.smithi079.stderr: 32550K .......... .......... .......... .......... .......... 44% 247M 0s 2019-12-11T16:46:06.470 INFO:teuthology.orchestra.run.smithi079.stderr: 32600K .......... .......... .......... .......... .......... 44% 216M 0s 2019-12-11T16:46:06.470 INFO:teuthology.orchestra.run.smithi079.stderr: 32650K .......... .......... .......... .......... .......... 44% 226M 0s 2019-12-11T16:46:06.470 INFO:teuthology.orchestra.run.smithi079.stderr: 32700K .......... .......... .......... .......... .......... 44% 220M 0s 2019-12-11T16:46:06.470 INFO:teuthology.orchestra.run.smithi079.stderr: 32750K .......... .......... .......... .......... .......... 44% 239M 0s 2019-12-11T16:46:06.470 INFO:teuthology.orchestra.run.smithi079.stderr: 32800K .......... .......... .......... .......... .......... 44% 278M 0s 2019-12-11T16:46:06.470 INFO:teuthology.orchestra.run.smithi079.stderr: 32850K .......... .......... .......... .......... .......... 44% 209M 0s 2019-12-11T16:46:06.470 INFO:teuthology.orchestra.run.smithi079.stderr: 32900K .......... .......... .......... .......... .......... 44% 219M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 32950K .......... .......... .......... .......... .......... 44% 216M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 33000K .......... .......... .......... .......... .......... 44% 286M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 33050K .......... .......... .......... .......... .......... 44% 249M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 33100K .......... .......... .......... .......... .......... 45% 210M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 33150K .......... .......... .......... .......... .......... 45% 213M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 33200K .......... .......... .......... .......... .......... 45% 284M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 33250K .......... .......... .......... .......... .......... 45% 254M 0s 2019-12-11T16:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr: 33300K .......... .......... .......... .......... .......... 45% 199M 0s 2019-12-11T16:46:06.472 INFO:teuthology.orchestra.run.smithi079.stderr: 33350K .......... .......... .......... .......... .......... 45% 253M 0s 2019-12-11T16:46:06.472 INFO:teuthology.orchestra.run.smithi079.stderr: 33400K .......... .......... .......... .......... .......... 45% 289M 0s 2019-12-11T16:46:06.472 INFO:teuthology.orchestra.run.smithi079.stderr: 33450K .......... .......... .......... .......... .......... 45% 286M 0s 2019-12-11T16:46:06.472 INFO:teuthology.orchestra.run.smithi079.stderr: 33500K .......... .......... .......... .......... .......... 45% 270M 0s 2019-12-11T16:46:06.472 INFO:teuthology.orchestra.run.smithi079.stderr: 33550K .......... .......... .......... .......... .......... 45% 178M 0s 2019-12-11T16:46:06.472 INFO:teuthology.orchestra.run.smithi079.stderr: 33600K .......... .......... .......... .......... .......... 45% 239M 0s 2019-12-11T16:46:06.473 INFO:teuthology.orchestra.run.smithi079.stderr: 33650K .......... .......... .......... .......... .......... 45% 281M 0s 2019-12-11T16:46:06.473 INFO:teuthology.orchestra.run.smithi079.stderr: 33700K .......... .......... .......... .......... .......... 45% 294M 0s 2019-12-11T16:46:06.473 INFO:teuthology.orchestra.run.smithi079.stderr: 33750K .......... .......... .......... .......... .......... 45% 244M 0s 2019-12-11T16:46:06.473 INFO:teuthology.orchestra.run.smithi079.stderr: 33800K .......... .......... .......... .......... .......... 46% 250M 0s 2019-12-11T16:46:06.473 INFO:teuthology.orchestra.run.smithi079.stderr: 33850K .......... .......... .......... .......... .......... 46% 234M 0s 2019-12-11T16:46:06.473 INFO:teuthology.orchestra.run.smithi079.stderr: 33900K .......... .......... .......... .......... .......... 46% 209M 0s 2019-12-11T16:46:06.474 INFO:teuthology.orchestra.run.smithi079.stderr: 33950K .......... .......... .......... .......... .......... 46% 217M 0s 2019-12-11T16:46:06.474 INFO:teuthology.orchestra.run.smithi079.stderr: 34000K .......... .......... .......... .......... .......... 46% 227M 0s 2019-12-11T16:46:06.474 INFO:teuthology.orchestra.run.smithi079.stderr: 34050K .......... .......... .......... .......... .......... 46% 238M 0s 2019-12-11T16:46:06.474 INFO:teuthology.orchestra.run.smithi079.stderr: 34100K .......... .......... .......... .......... .......... 46% 233M 0s 2019-12-11T16:46:06.474 INFO:teuthology.orchestra.run.smithi079.stderr: 34150K .......... .......... .......... .......... .......... 46% 212M 0s 2019-12-11T16:46:06.474 INFO:teuthology.orchestra.run.smithi079.stderr: 34200K .......... .......... .......... .......... .......... 46% 242M 0s 2019-12-11T16:46:06.474 INFO:teuthology.orchestra.run.smithi079.stderr: 34250K .......... .......... .......... .......... .......... 46% 255M 0s 2019-12-11T16:46:06.475 INFO:teuthology.orchestra.run.smithi079.stderr: 34300K .......... .......... .......... .......... .......... 46% 215M 0s 2019-12-11T16:46:06.475 INFO:teuthology.orchestra.run.smithi079.stderr: 34350K .......... .......... .......... .......... .......... 46% 222M 0s 2019-12-11T16:46:06.475 INFO:teuthology.orchestra.run.smithi079.stderr: 34400K .......... .......... .......... .......... .......... 46% 224M 0s 2019-12-11T16:46:06.475 INFO:teuthology.orchestra.run.smithi079.stderr: 34450K .......... .......... .......... .......... .......... 46% 238M 0s 2019-12-11T16:46:06.475 INFO:teuthology.orchestra.run.smithi079.stderr: 34500K .......... .......... .......... .......... .......... 46% 251M 0s 2019-12-11T16:46:06.475 INFO:teuthology.orchestra.run.smithi079.stderr: 34550K .......... .......... .......... .......... .......... 47% 221M 0s 2019-12-11T16:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: 34600K .......... .......... .......... .......... .......... 47% 288M 0s 2019-12-11T16:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: 34650K .......... .......... .......... .......... .......... 47% 238M 0s 2019-12-11T16:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: 34700K .......... .......... .......... .......... .......... 47% 222M 0s 2019-12-11T16:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: 34750K .......... .......... .......... .......... .......... 47% 204M 0s 2019-12-11T16:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: 34800K .......... .......... .......... .......... .......... 47% 244M 0s 2019-12-11T16:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: 34850K .......... .......... .......... .......... .......... 47% 290M 0s 2019-12-11T16:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: 34900K .......... .......... .......... .......... .......... 47% 252M 0s 2019-12-11T16:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: 34950K .......... .......... .......... .......... .......... 47% 185M 0s 2019-12-11T16:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: 35000K .......... .......... .......... .......... .......... 47% 248M 0s 2019-12-11T16:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: 35050K .......... .......... .......... .......... .......... 47% 223M 0s 2019-12-11T16:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: 35100K .......... .......... .......... .......... .......... 47% 220M 0s 2019-12-11T16:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: 35150K .......... .......... .......... .......... .......... 47% 217M 0s 2019-12-11T16:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: 35200K .......... .......... .......... .......... .......... 47% 293M 0s 2019-12-11T16:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: 35250K .......... .......... .......... .......... .......... 47% 248M 0s 2019-12-11T16:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr: 35300K .......... .......... .......... .......... .......... 48% 251M 0s 2019-12-11T16:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr: 35350K .......... .......... .......... .......... .......... 48% 197M 0s 2019-12-11T16:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr: 35400K .......... .......... .......... .......... .......... 48% 239M 0s 2019-12-11T16:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr: 35450K .......... .......... .......... .......... .......... 48% 245M 0s 2019-12-11T16:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr: 35500K .......... .......... .......... .......... .......... 48% 267M 0s 2019-12-11T16:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr: 35550K .......... .......... .......... .......... .......... 48% 213M 0s 2019-12-11T16:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr: 35600K .......... .......... .......... .......... .......... 48% 240M 0s 2019-12-11T16:46:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: 35650K .......... .......... .......... .......... .......... 48% 239M 0s 2019-12-11T16:46:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: 35700K .......... .......... .......... .......... .......... 48% 249M 0s 2019-12-11T16:46:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: 35750K .......... .......... .......... .......... .......... 48% 234M 0s 2019-12-11T16:46:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: 35800K .......... .......... .......... .......... .......... 48% 241M 0s 2019-12-11T16:46:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: 35850K .......... .......... .......... .......... .......... 48% 253M 0s 2019-12-11T16:46:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: 35900K .......... .......... .......... .......... .......... 48% 274M 0s 2019-12-11T16:46:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: 35950K .......... .......... .......... .......... .......... 48% 169M 0s 2019-12-11T16:46:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 36000K .......... .......... .......... .......... .......... 49% 274M 0s 2019-12-11T16:46:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 36050K .......... .......... .......... .......... .......... 49% 197M 0s 2019-12-11T16:46:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 36100K .......... .......... .......... .......... .......... 49% 241M 0s 2019-12-11T16:46:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 36150K .......... .......... .......... .......... .......... 49% 261M 0s 2019-12-11T16:46:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 36200K .......... .......... .......... .......... .......... 49% 216M 0s 2019-12-11T16:46:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 36250K .......... .......... .......... .......... .......... 49% 235M 0s 2019-12-11T16:46:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 36300K .......... .......... .......... .......... .......... 49% 249M 0s 2019-12-11T16:46:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: 36350K .......... .......... .......... .......... .......... 49% 178M 0s 2019-12-11T16:46:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: 36400K .......... .......... .......... .......... .......... 49% 291M 0s 2019-12-11T16:46:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: 36450K .......... .......... .......... .......... .......... 49% 182M 0s 2019-12-11T16:46:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: 36500K .......... .......... .......... .......... .......... 49% 213M 0s 2019-12-11T16:46:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: 36550K .......... .......... .......... .......... .......... 49% 203M 0s 2019-12-11T16:46:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: 36600K .......... .......... .......... .......... .......... 49% 278M 0s 2019-12-11T16:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 36650K .......... .......... .......... .......... .......... 49% 233M 0s 2019-12-11T16:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 36700K .......... .......... .......... .......... .......... 49% 205M 0s 2019-12-11T16:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 36750K .......... .......... .......... .......... .......... 50% 184M 0s 2019-12-11T16:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 36800K .......... .......... .......... .......... .......... 50% 231M 0s 2019-12-11T16:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 36850K .......... .......... .......... .......... .......... 50% 251M 0s 2019-12-11T16:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 36900K .......... .......... .......... .......... .......... 50% 239M 0s 2019-12-11T16:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 36950K .......... .......... .......... .......... .......... 50% 245M 0s 2019-12-11T16:46:06.483 INFO:teuthology.orchestra.run.smithi079.stderr: 37000K .......... .......... .......... .......... .......... 50% 241M 0s 2019-12-11T16:46:06.483 INFO:teuthology.orchestra.run.smithi079.stderr: 37050K .......... .......... .......... .......... .......... 50% 216M 0s 2019-12-11T16:46:06.483 INFO:teuthology.orchestra.run.smithi079.stderr: 37100K .......... .......... .......... .......... .......... 50% 254M 0s 2019-12-11T16:46:06.483 INFO:teuthology.orchestra.run.smithi079.stderr: 37150K .......... .......... .......... .......... .......... 50% 204M 0s 2019-12-11T16:46:06.483 INFO:teuthology.orchestra.run.smithi079.stderr: 37200K .......... .......... .......... .......... .......... 50% 227M 0s 2019-12-11T16:46:06.483 INFO:teuthology.orchestra.run.smithi079.stderr: 37250K .......... .......... .......... .......... .......... 50% 257M 0s 2019-12-11T16:46:06.483 INFO:teuthology.orchestra.run.smithi079.stderr: 37300K .......... .......... .......... .......... .......... 50% 225M 0s 2019-12-11T16:46:06.484 INFO:teuthology.orchestra.run.smithi079.stderr: 37350K .......... .......... .......... .......... .......... 50% 178M 0s 2019-12-11T16:46:06.484 INFO:teuthology.orchestra.run.smithi079.stderr: 37400K .......... .......... .......... .......... .......... 50% 192M 0s 2019-12-11T16:46:06.484 INFO:teuthology.orchestra.run.smithi079.stderr: 37450K .......... .......... .......... .......... .......... 50% 212M 0s 2019-12-11T16:46:06.484 INFO:teuthology.orchestra.run.smithi079.stderr: 37500K .......... .......... .......... .......... .......... 51% 222M 0s 2019-12-11T16:46:06.484 INFO:teuthology.orchestra.run.smithi079.stderr: 37550K .......... .......... .......... .......... .......... 51% 199M 0s 2019-12-11T16:46:06.484 INFO:teuthology.orchestra.run.smithi079.stderr: 37600K .......... .......... .......... .......... .......... 51% 252M 0s 2019-12-11T16:46:06.484 INFO:teuthology.orchestra.run.smithi079.stderr: 37650K .......... .......... .......... .......... .......... 51% 225M 0s 2019-12-11T16:46:06.485 INFO:teuthology.orchestra.run.smithi079.stderr: 37700K .......... .......... .......... .......... .......... 51% 200M 0s 2019-12-11T16:46:06.485 INFO:teuthology.orchestra.run.smithi079.stderr: 37750K .......... .......... .......... .......... .......... 51% 213M 0s 2019-12-11T16:46:06.485 INFO:teuthology.orchestra.run.smithi079.stderr: 37800K .......... .......... .......... .......... .......... 51% 230M 0s 2019-12-11T16:46:06.485 INFO:teuthology.orchestra.run.smithi079.stderr: 37850K .......... .......... .......... .......... .......... 51% 278M 0s 2019-12-11T16:46:06.485 INFO:teuthology.orchestra.run.smithi079.stderr: 37900K .......... .......... .......... .......... .......... 51% 262M 0s 2019-12-11T16:46:06.485 INFO:teuthology.orchestra.run.smithi079.stderr: 37950K .......... .......... .......... .......... .......... 51% 176M 0s 2019-12-11T16:46:06.485 INFO:teuthology.orchestra.run.smithi079.stderr: 38000K .......... .......... .......... .......... .......... 51% 204M 0s 2019-12-11T16:46:06.486 INFO:teuthology.orchestra.run.smithi079.stderr: 38050K .......... .......... .......... .......... .......... 51% 236M 0s 2019-12-11T16:46:06.486 INFO:teuthology.orchestra.run.smithi079.stderr: 38100K .......... .......... .......... .......... .......... 51% 257M 0s 2019-12-11T16:46:06.486 INFO:teuthology.orchestra.run.smithi079.stderr: 38150K .......... .......... .......... .......... .......... 51% 197M 0s 2019-12-11T16:46:06.486 INFO:teuthology.orchestra.run.smithi079.stderr: 38200K .......... .......... .......... .......... .......... 51% 243M 0s 2019-12-11T16:46:06.486 INFO:teuthology.orchestra.run.smithi079.stderr: 38250K .......... .......... .......... .......... .......... 52% 229M 0s 2019-12-11T16:46:06.486 INFO:teuthology.orchestra.run.smithi079.stderr: 38300K .......... .......... .......... .......... .......... 52% 242M 0s 2019-12-11T16:46:06.486 INFO:teuthology.orchestra.run.smithi079.stderr: 38350K .......... .......... .......... .......... .......... 52% 188M 0s 2019-12-11T16:46:06.487 INFO:teuthology.orchestra.run.smithi079.stderr: 38400K .......... .......... .......... .......... .......... 52% 242M 0s 2019-12-11T16:46:06.487 INFO:teuthology.orchestra.run.smithi079.stderr: 38450K .......... .......... .......... .......... .......... 52% 214M 0s 2019-12-11T16:46:06.487 INFO:teuthology.orchestra.run.smithi079.stderr: 38500K .......... .......... .......... .......... .......... 52% 238M 0s 2019-12-11T16:46:06.487 INFO:teuthology.orchestra.run.smithi079.stderr: 38550K .......... .......... .......... .......... .......... 52% 234M 0s 2019-12-11T16:46:06.487 INFO:teuthology.orchestra.run.smithi079.stderr: 38600K .......... .......... .......... .......... .......... 52% 221M 0s 2019-12-11T16:46:06.487 INFO:teuthology.orchestra.run.smithi079.stderr: 38650K .......... .......... .......... .......... .......... 52% 256M 0s 2019-12-11T16:46:06.487 INFO:teuthology.orchestra.run.smithi079.stderr: 38700K .......... .......... .......... .......... .......... 52% 242M 0s 2019-12-11T16:46:06.488 INFO:teuthology.orchestra.run.smithi079.stderr: 38750K .......... .......... .......... .......... .......... 52% 176M 0s 2019-12-11T16:46:06.488 INFO:teuthology.orchestra.run.smithi079.stderr: 38800K .......... .......... .......... .......... .......... 52% 264M 0s 2019-12-11T16:46:06.488 INFO:teuthology.orchestra.run.smithi079.stderr: 38850K .......... .......... .......... .......... .......... 52% 227M 0s 2019-12-11T16:46:06.488 INFO:teuthology.orchestra.run.smithi079.stderr: 38900K .......... .......... .......... .......... .......... 52% 262M 0s 2019-12-11T16:46:06.488 INFO:teuthology.orchestra.run.smithi079.stderr: 38950K .......... .......... .......... .......... .......... 53% 192M 0s 2019-12-11T16:46:06.488 INFO:teuthology.orchestra.run.smithi079.stderr: 39000K .......... .......... .......... .......... .......... 53% 215M 0s 2019-12-11T16:46:06.488 INFO:teuthology.orchestra.run.smithi079.stderr: 39050K .......... .......... .......... .......... .......... 53% 230M 0s 2019-12-11T16:46:06.489 INFO:teuthology.orchestra.run.smithi079.stderr: 39100K .......... .......... .......... .......... .......... 53% 262M 0s 2019-12-11T16:46:06.489 INFO:teuthology.orchestra.run.smithi079.stderr: 39150K .......... .......... .......... .......... .......... 53% 177M 0s 2019-12-11T16:46:06.489 INFO:teuthology.orchestra.run.smithi079.stderr: 39200K .......... .......... .......... .......... .......... 53% 282M 0s 2019-12-11T16:46:06.489 INFO:teuthology.orchestra.run.smithi079.stderr: 39250K .......... .......... .......... .......... .......... 53% 276M 0s 2019-12-11T16:46:06.489 INFO:teuthology.orchestra.run.smithi079.stderr: 39300K .......... .......... .......... .......... .......... 53% 206M 0s 2019-12-11T16:46:06.489 INFO:teuthology.orchestra.run.smithi079.stderr: 39350K .......... .......... .......... .......... .......... 53% 249M 0s 2019-12-11T16:46:06.490 INFO:teuthology.orchestra.run.smithi079.stderr: 39400K .......... .......... .......... .......... .......... 53% 243M 0s 2019-12-11T16:46:06.490 INFO:teuthology.orchestra.run.smithi079.stderr: 39450K .......... .......... .......... .......... .......... 53% 282M 0s 2019-12-11T16:46:06.490 INFO:teuthology.orchestra.run.smithi079.stderr: 39500K .......... .......... .......... .......... .......... 53% 367M 0s 2019-12-11T16:46:06.490 INFO:teuthology.orchestra.run.smithi079.stderr: 39550K .......... .......... .......... .......... .......... 53% 304M 0s 2019-12-11T16:46:06.490 INFO:teuthology.orchestra.run.smithi079.stderr: 39600K .......... .......... .......... .......... .......... 53% 351M 0s 2019-12-11T16:46:06.490 INFO:teuthology.orchestra.run.smithi079.stderr: 39650K .......... .......... .......... .......... .......... 53% 406M 0s 2019-12-11T16:46:06.490 INFO:teuthology.orchestra.run.smithi079.stderr: 39700K .......... .......... .......... .......... .......... 54% 388M 0s 2019-12-11T16:46:06.491 INFO:teuthology.orchestra.run.smithi079.stderr: 39750K .......... .......... .......... .......... .......... 54% 303M 0s 2019-12-11T16:46:06.491 INFO:teuthology.orchestra.run.smithi079.stderr: 39800K .......... .......... .......... .......... .......... 54% 366M 0s 2019-12-11T16:46:06.491 INFO:teuthology.orchestra.run.smithi079.stderr: 39850K .......... .......... .......... .......... .......... 54% 404M 0s 2019-12-11T16:46:06.491 INFO:teuthology.orchestra.run.smithi079.stderr: 39900K .......... .......... .......... .......... .......... 54% 344M 0s 2019-12-11T16:46:06.491 INFO:teuthology.orchestra.run.smithi079.stderr: 39950K .......... .......... .......... .......... .......... 54% 228M 0s 2019-12-11T16:46:06.491 INFO:teuthology.orchestra.run.smithi079.stderr: 40000K .......... .......... .......... .......... .......... 54% 405M 0s 2019-12-11T16:46:06.492 INFO:teuthology.orchestra.run.smithi079.stderr: 40050K .......... .......... .......... .......... .......... 54% 307M 0s 2019-12-11T16:46:06.492 INFO:teuthology.orchestra.run.smithi079.stderr: 40100K .......... .......... .......... .......... .......... 54% 338M 0s 2019-12-11T16:46:06.492 INFO:teuthology.orchestra.run.smithi079.stderr: 40150K .......... .......... .......... .......... .......... 54% 309M 0s 2019-12-11T16:46:06.492 INFO:teuthology.orchestra.run.smithi079.stderr: 40200K .......... .......... .......... .......... .......... 54% 392M 0s 2019-12-11T16:46:06.492 INFO:teuthology.orchestra.run.smithi079.stderr: 40250K .......... .......... .......... .......... .......... 54% 345M 0s 2019-12-11T16:46:06.492 INFO:teuthology.orchestra.run.smithi079.stderr: 40300K .......... .......... .......... .......... .......... 54% 299M 0s 2019-12-11T16:46:06.492 INFO:teuthology.orchestra.run.smithi079.stderr: 40350K .......... .......... .......... .......... .......... 54% 316M 0s 2019-12-11T16:46:06.493 INFO:teuthology.orchestra.run.smithi079.stderr: 40400K .......... .......... .......... .......... .......... 54% 311M 0s 2019-12-11T16:46:06.493 INFO:teuthology.orchestra.run.smithi079.stderr: 40450K .......... .......... .......... .......... .......... 55% 332M 0s 2019-12-11T16:46:06.493 INFO:teuthology.orchestra.run.smithi079.stderr: 40500K .......... .......... .......... .......... .......... 55% 368M 0s 2019-12-11T16:46:06.493 INFO:teuthology.orchestra.run.smithi079.stderr: 40550K .......... .......... .......... .......... .......... 55% 324M 0s 2019-12-11T16:46:06.493 INFO:teuthology.orchestra.run.smithi079.stderr: 40600K .......... .......... .......... .......... .......... 55% 310M 0s 2019-12-11T16:46:06.493 INFO:teuthology.orchestra.run.smithi079.stderr: 40650K .......... .......... .......... .......... .......... 55% 367M 0s 2019-12-11T16:46:06.493 INFO:teuthology.orchestra.run.smithi079.stderr: 40700K .......... .......... .......... .......... .......... 55% 379M 0s 2019-12-11T16:46:06.494 INFO:teuthology.orchestra.run.smithi079.stderr: 40750K .......... .......... .......... .......... .......... 55% 252M 0s 2019-12-11T16:46:06.494 INFO:teuthology.orchestra.run.smithi079.stderr: 40800K .......... .......... .......... .......... .......... 55% 364M 0s 2019-12-11T16:46:06.494 INFO:teuthology.orchestra.run.smithi079.stderr: 40850K .......... .......... .......... .......... .......... 55% 321M 0s 2019-12-11T16:46:06.494 INFO:teuthology.orchestra.run.smithi079.stderr: 40900K .......... .......... .......... .......... .......... 55% 403M 0s 2019-12-11T16:46:06.494 INFO:teuthology.orchestra.run.smithi079.stderr: 40950K .......... .......... .......... .......... .......... 55% 343M 0s 2019-12-11T16:46:06.494 INFO:teuthology.orchestra.run.smithi079.stderr: 41000K .......... .......... .......... .......... .......... 55% 324M 0s 2019-12-11T16:46:06.494 INFO:teuthology.orchestra.run.smithi079.stderr: 41050K .......... .......... .......... .......... .......... 55% 377M 0s 2019-12-11T16:46:06.495 INFO:teuthology.orchestra.run.smithi079.stderr: 41100K .......... .......... .......... .......... .......... 55% 329M 0s 2019-12-11T16:46:06.495 INFO:teuthology.orchestra.run.smithi079.stderr: 41150K .......... .......... .......... .......... .......... 56% 272M 0s 2019-12-11T16:46:06.495 INFO:teuthology.orchestra.run.smithi079.stderr: 41200K .......... .......... .......... .......... .......... 56% 389M 0s 2019-12-11T16:46:06.495 INFO:teuthology.orchestra.run.smithi079.stderr: 41250K .......... .......... .......... .......... .......... 56% 334M 0s 2019-12-11T16:46:06.495 INFO:teuthology.orchestra.run.smithi079.stderr: 41300K .......... .......... .......... .......... .......... 56% 327M 0s 2019-12-11T16:46:06.495 INFO:teuthology.orchestra.run.smithi079.stderr: 41350K .......... .......... .......... .......... .......... 56% 350M 0s 2019-12-11T16:46:06.495 INFO:teuthology.orchestra.run.smithi079.stderr: 41400K .......... .......... .......... .......... .......... 56% 341M 0s 2019-12-11T16:46:06.496 INFO:teuthology.orchestra.run.smithi079.stderr: 41450K .......... .......... .......... .......... .......... 56% 329M 0s 2019-12-11T16:46:06.496 INFO:teuthology.orchestra.run.smithi079.stderr: 41500K .......... .......... .......... .......... .......... 56% 373M 0s 2019-12-11T16:46:06.496 INFO:teuthology.orchestra.run.smithi079.stderr: 41550K .......... .......... .......... .......... .......... 56% 304M 0s 2019-12-11T16:46:06.496 INFO:teuthology.orchestra.run.smithi079.stderr: 41600K .......... .......... .......... .......... .......... 56% 343M 0s 2019-12-11T16:46:06.496 INFO:teuthology.orchestra.run.smithi079.stderr: 41650K .......... .......... .......... .......... .......... 56% 340M 0s 2019-12-11T16:46:06.496 INFO:teuthology.orchestra.run.smithi079.stderr: 41700K .......... .......... .......... .......... .......... 56% 382M 0s 2019-12-11T16:46:06.496 INFO:teuthology.orchestra.run.smithi079.stderr: 41750K .......... .......... .......... .......... .......... 56% 340M 0s 2019-12-11T16:46:06.497 INFO:teuthology.orchestra.run.smithi079.stderr: 41800K .......... .......... .......... .......... .......... 56% 363M 0s 2019-12-11T16:46:06.497 INFO:teuthology.orchestra.run.smithi079.stderr: 41850K .......... .......... .......... .......... .......... 56% 277M 0s 2019-12-11T16:46:06.497 INFO:teuthology.orchestra.run.smithi079.stderr: 41900K .......... .......... .......... .......... .......... 57% 308M 0s 2019-12-11T16:46:06.497 INFO:teuthology.orchestra.run.smithi079.stderr: 41950K .......... .......... .......... .......... .......... 57% 332M 0s 2019-12-11T16:46:06.497 INFO:teuthology.orchestra.run.smithi079.stderr: 42000K .......... .......... .......... .......... .......... 57% 288M 0s 2019-12-11T16:46:06.497 INFO:teuthology.orchestra.run.smithi079.stderr: 42050K .......... .......... .......... .......... .......... 57% 343M 0s 2019-12-11T16:46:06.498 INFO:teuthology.orchestra.run.smithi079.stderr: 42100K .......... .......... .......... .......... .......... 57% 346M 0s 2019-12-11T16:46:06.498 INFO:teuthology.orchestra.run.smithi079.stderr: 42150K .......... .......... .......... .......... .......... 57% 255M 0s 2019-12-11T16:46:06.498 INFO:teuthology.orchestra.run.smithi079.stderr: 42200K .......... .......... .......... .......... .......... 57% 379M 0s 2019-12-11T16:46:06.498 INFO:teuthology.orchestra.run.smithi079.stderr: 42250K .......... .......... .......... .......... .......... 57% 417M 0s 2019-12-11T16:46:06.498 INFO:teuthology.orchestra.run.smithi079.stderr: 42300K .......... .......... .......... .......... .......... 57% 389M 0s 2019-12-11T16:46:06.498 INFO:teuthology.orchestra.run.smithi079.stderr: 42350K .......... .......... .......... .......... .......... 57% 263M 0s 2019-12-11T16:46:06.498 INFO:teuthology.orchestra.run.smithi079.stderr: 42400K .......... .......... .......... .......... .......... 57% 319M 0s 2019-12-11T16:46:06.499 INFO:teuthology.orchestra.run.smithi079.stderr: 42450K .......... .......... .......... .......... .......... 57% 349M 0s 2019-12-11T16:46:06.499 INFO:teuthology.orchestra.run.smithi079.stderr: 42500K .......... .......... .......... .......... .......... 57% 356M 0s 2019-12-11T16:46:06.499 INFO:teuthology.orchestra.run.smithi079.stderr: 42550K .......... .......... .......... .......... .......... 57% 302M 0s 2019-12-11T16:46:06.499 INFO:teuthology.orchestra.run.smithi079.stderr: 42600K .......... .......... .......... .......... .......... 57% 260M 0s 2019-12-11T16:46:06.499 INFO:teuthology.orchestra.run.smithi079.stderr: 42650K .......... .......... .......... .......... .......... 58% 330M 0s 2019-12-11T16:46:06.499 INFO:teuthology.orchestra.run.smithi079.stderr: 42700K .......... .......... .......... .......... .......... 58% 303M 0s 2019-12-11T16:46:06.500 INFO:teuthology.orchestra.run.smithi079.stderr: 42750K .......... .......... .......... .......... .......... 58% 263M 0s 2019-12-11T16:46:06.500 INFO:teuthology.orchestra.run.smithi079.stderr: 42800K .......... .......... .......... .......... .......... 58% 309M 0s 2019-12-11T16:46:06.500 INFO:teuthology.orchestra.run.smithi079.stderr: 42850K .......... .......... .......... .......... .......... 58% 369M 0s 2019-12-11T16:46:06.500 INFO:teuthology.orchestra.run.smithi079.stderr: 42900K .......... .......... .......... .......... .......... 58% 333M 0s 2019-12-11T16:46:06.500 INFO:teuthology.orchestra.run.smithi079.stderr: 42950K .......... .......... .......... .......... .......... 58% 298M 0s 2019-12-11T16:46:06.500 INFO:teuthology.orchestra.run.smithi079.stderr: 43000K .......... .......... .......... .......... .......... 58% 335M 0s 2019-12-11T16:46:06.501 INFO:teuthology.orchestra.run.smithi079.stderr: 43050K .......... .......... .......... .......... .......... 58% 379M 0s 2019-12-11T16:46:06.501 INFO:teuthology.orchestra.run.smithi079.stderr: 43100K .......... .......... .......... .......... .......... 58% 327M 0s 2019-12-11T16:46:06.501 INFO:teuthology.orchestra.run.smithi079.stderr: 43150K .......... .......... .......... .......... .......... 58% 282M 0s 2019-12-11T16:46:06.501 INFO:teuthology.orchestra.run.smithi079.stderr: 43200K .......... .......... .......... .......... .......... 58% 356M 0s 2019-12-11T16:46:06.501 INFO:teuthology.orchestra.run.smithi079.stderr: 43250K .......... .......... .......... .......... .......... 58% 361M 0s 2019-12-11T16:46:06.501 INFO:teuthology.orchestra.run.smithi079.stderr: 43300K .......... .......... .......... .......... .......... 58% 319M 0s 2019-12-11T16:46:06.502 INFO:teuthology.orchestra.run.smithi079.stderr: 43350K .......... .......... .......... .......... .......... 59% 288M 0s 2019-12-11T16:46:06.502 INFO:teuthology.orchestra.run.smithi079.stderr: 43400K .......... .......... .......... .......... .......... 59% 378M 0s 2019-12-11T16:46:06.502 INFO:teuthology.orchestra.run.smithi079.stderr: 43450K .......... .......... .......... .......... .......... 59% 344M 0s 2019-12-11T16:46:06.502 INFO:teuthology.orchestra.run.smithi079.stderr: 43500K .......... .......... .......... .......... .......... 59% 365M 0s 2019-12-11T16:46:06.502 INFO:teuthology.orchestra.run.smithi079.stderr: 43550K .......... .......... .......... .......... .......... 59% 273M 0s 2019-12-11T16:46:06.502 INFO:teuthology.orchestra.run.smithi079.stderr: 43600K .......... .......... .......... .......... .......... 59% 329M 0s 2019-12-11T16:46:06.502 INFO:teuthology.orchestra.run.smithi079.stderr: 43650K .......... .......... .......... .......... .......... 59% 337M 0s 2019-12-11T16:46:06.503 INFO:teuthology.orchestra.run.smithi079.stderr: 43700K .......... .......... .......... .......... .......... 59% 332M 0s 2019-12-11T16:46:06.503 INFO:teuthology.orchestra.run.smithi079.stderr: 43750K .......... .......... .......... .......... .......... 59% 269M 0s 2019-12-11T16:46:06.503 INFO:teuthology.orchestra.run.smithi079.stderr: 43800K .......... .......... .......... .......... .......... 59% 294M 0s 2019-12-11T16:46:06.503 INFO:teuthology.orchestra.run.smithi079.stderr: 43850K .......... .......... .......... .......... .......... 59% 307M 0s 2019-12-11T16:46:06.503 INFO:teuthology.orchestra.run.smithi079.stderr: 43900K .......... .......... .......... .......... .......... 59% 275M 0s 2019-12-11T16:46:06.503 INFO:teuthology.orchestra.run.smithi079.stderr: 43950K .......... .......... .......... .......... .......... 59% 258M 0s 2019-12-11T16:46:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 44000K .......... .......... .......... .......... .......... 59% 341M 0s 2019-12-11T16:46:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 44050K .......... .......... .......... .......... .......... 59% 305M 0s 2019-12-11T16:46:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 44100K .......... .......... .......... .......... .......... 60% 320M 0s 2019-12-11T16:46:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 44150K .......... .......... .......... .......... .......... 60% 353M 0s 2019-12-11T16:46:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 44200K .......... .......... .......... .......... .......... 60% 391M 0s 2019-12-11T16:46:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 44250K .......... .......... .......... .......... .......... 60% 367M 0s 2019-12-11T16:46:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 44300K .......... .......... .......... .......... .......... 60% 297M 0s 2019-12-11T16:46:06.505 INFO:teuthology.orchestra.run.smithi079.stderr: 44350K .......... .......... .......... .......... .......... 60% 283M 0s 2019-12-11T16:46:06.505 INFO:teuthology.orchestra.run.smithi079.stderr: 44400K .......... .......... .......... .......... .......... 60% 322M 0s 2019-12-11T16:46:06.505 INFO:teuthology.orchestra.run.smithi079.stderr: 44450K .......... .......... .......... .......... .......... 60% 307M 0s 2019-12-11T16:46:06.505 INFO:teuthology.orchestra.run.smithi079.stderr: 44500K .......... .......... .......... .......... .......... 60% 304M 0s 2019-12-11T16:46:06.505 INFO:teuthology.orchestra.run.smithi079.stderr: 44550K .......... .......... .......... .......... .......... 60% 296M 0s 2019-12-11T16:46:06.505 INFO:teuthology.orchestra.run.smithi079.stderr: 44600K .......... .......... .......... .......... .......... 60% 352M 0s 2019-12-11T16:46:06.506 INFO:teuthology.orchestra.run.smithi079.stderr: 44650K .......... .......... .......... .......... .......... 60% 317M 0s 2019-12-11T16:46:06.506 INFO:teuthology.orchestra.run.smithi079.stderr: 44700K .......... .......... .......... .......... .......... 60% 375M 0s 2019-12-11T16:46:06.506 INFO:teuthology.orchestra.run.smithi079.stderr: 44750K .......... .......... .......... .......... .......... 60% 268M 0s 2019-12-11T16:46:06.506 INFO:teuthology.orchestra.run.smithi079.stderr: 44800K .......... .......... .......... .......... .......... 60% 342M 0s 2019-12-11T16:46:06.506 INFO:teuthology.orchestra.run.smithi079.stderr: 44850K .......... .......... .......... .......... .......... 61% 288M 0s 2019-12-11T16:46:06.506 INFO:teuthology.orchestra.run.smithi079.stderr: 44900K .......... .......... .......... .......... .......... 61% 306M 0s 2019-12-11T16:46:06.506 INFO:teuthology.orchestra.run.smithi079.stderr: 44950K .......... .......... .......... .......... .......... 61% 257M 0s 2019-12-11T16:46:06.507 INFO:teuthology.orchestra.run.smithi079.stderr: 45000K .......... .......... .......... .......... .......... 61% 343M 0s 2019-12-11T16:46:06.507 INFO:teuthology.orchestra.run.smithi079.stderr: 45050K .......... .......... .......... .......... .......... 61% 327M 0s 2019-12-11T16:46:06.507 INFO:teuthology.orchestra.run.smithi079.stderr: 45100K .......... .......... .......... .......... .......... 61% 307M 0s 2019-12-11T16:46:06.507 INFO:teuthology.orchestra.run.smithi079.stderr: 45150K .......... .......... .......... .......... .......... 61% 289M 0s 2019-12-11T16:46:06.507 INFO:teuthology.orchestra.run.smithi079.stderr: 45200K .......... .......... .......... .......... .......... 61% 293M 0s 2019-12-11T16:46:06.507 INFO:teuthology.orchestra.run.smithi079.stderr: 45250K .......... .......... .......... .......... .......... 61% 394M 0s 2019-12-11T16:46:06.507 INFO:teuthology.orchestra.run.smithi079.stderr: 45300K .......... .......... .......... .......... .......... 61% 333M 0s 2019-12-11T16:46:06.508 INFO:teuthology.orchestra.run.smithi079.stderr: 45350K .......... .......... .......... .......... .......... 61% 278M 0s 2019-12-11T16:46:06.508 INFO:teuthology.orchestra.run.smithi079.stderr: 45400K .......... .......... .......... .......... .......... 61% 322M 0s 2019-12-11T16:46:06.508 INFO:teuthology.orchestra.run.smithi079.stderr: 45450K .......... .......... .......... .......... .......... 61% 367M 0s 2019-12-11T16:46:06.508 INFO:teuthology.orchestra.run.smithi079.stderr: 45500K .......... .......... .......... .......... .......... 61% 313M 0s 2019-12-11T16:46:06.508 INFO:teuthology.orchestra.run.smithi079.stderr: 45550K .......... .......... .......... .......... .......... 61% 291M 0s 2019-12-11T16:46:06.508 INFO:teuthology.orchestra.run.smithi079.stderr: 45600K .......... .......... .......... .......... .......... 62% 343M 0s 2019-12-11T16:46:06.508 INFO:teuthology.orchestra.run.smithi079.stderr: 45650K .......... .......... .......... .......... .......... 62% 346M 0s 2019-12-11T16:46:06.509 INFO:teuthology.orchestra.run.smithi079.stderr: 45700K .......... .......... .......... .......... .......... 62% 316M 0s 2019-12-11T16:46:06.509 INFO:teuthology.orchestra.run.smithi079.stderr: 45750K .......... .......... .......... .......... .......... 62% 312M 0s 2019-12-11T16:46:06.509 INFO:teuthology.orchestra.run.smithi079.stderr: 45800K .......... .......... .......... .......... .......... 62% 354M 0s 2019-12-11T16:46:06.509 INFO:teuthology.orchestra.run.smithi079.stderr: 45850K .......... .......... .......... .......... .......... 62% 392M 0s 2019-12-11T16:46:06.509 INFO:teuthology.orchestra.run.smithi079.stderr: 45900K .......... .......... .......... .......... .......... 62% 305M 0s 2019-12-11T16:46:06.509 INFO:teuthology.orchestra.run.smithi079.stderr: 45950K .......... .......... .......... .......... .......... 62% 284M 0s 2019-12-11T16:46:06.510 INFO:teuthology.orchestra.run.smithi079.stderr: 46000K .......... .......... .......... .......... .......... 62% 364M 0s 2019-12-11T16:46:06.510 INFO:teuthology.orchestra.run.smithi079.stderr: 46050K .......... .......... .......... .......... .......... 62% 326M 0s 2019-12-11T16:46:06.510 INFO:teuthology.orchestra.run.smithi079.stderr: 46100K .......... .......... .......... .......... .......... 62% 317M 0s 2019-12-11T16:46:06.510 INFO:teuthology.orchestra.run.smithi079.stderr: 46150K .......... .......... .......... .......... .......... 62% 278M 0s 2019-12-11T16:46:06.510 INFO:teuthology.orchestra.run.smithi079.stderr: 46200K .......... .......... .......... .......... .......... 62% 349M 0s 2019-12-11T16:46:06.510 INFO:teuthology.orchestra.run.smithi079.stderr: 46250K .......... .......... .......... .......... .......... 62% 317M 0s 2019-12-11T16:46:06.510 INFO:teuthology.orchestra.run.smithi079.stderr: 46300K .......... .......... .......... .......... .......... 63% 348M 0s 2019-12-11T16:46:06.511 INFO:teuthology.orchestra.run.smithi079.stderr: 46350K .......... .......... .......... .......... .......... 63% 247M 0s 2019-12-11T16:46:06.511 INFO:teuthology.orchestra.run.smithi079.stderr: 46400K .......... .......... .......... .......... .......... 63% 284M 0s 2019-12-11T16:46:06.511 INFO:teuthology.orchestra.run.smithi079.stderr: 46450K .......... .......... .......... .......... .......... 63% 312M 0s 2019-12-11T16:46:06.511 INFO:teuthology.orchestra.run.smithi079.stderr: 46500K .......... .......... .......... .......... .......... 63% 343M 0s 2019-12-11T16:46:06.511 INFO:teuthology.orchestra.run.smithi079.stderr: 46550K .......... .......... .......... .......... .......... 63% 274M 0s 2019-12-11T16:46:06.512 INFO:teuthology.orchestra.run.smithi079.stderr: 46600K .......... .......... .......... .......... .......... 63% 399M 0s 2019-12-11T16:46:06.512 INFO:teuthology.orchestra.run.smithi079.stderr: 46650K .......... .......... .......... .......... .......... 63% 353M 0s 2019-12-11T16:46:06.512 INFO:teuthology.orchestra.run.smithi079.stderr: 46700K .......... .......... .......... .......... .......... 63% 294M 0s 2019-12-11T16:46:06.512 INFO:teuthology.orchestra.run.smithi079.stderr: 46750K .......... .......... .......... .......... .......... 63% 326M 0s 2019-12-11T16:46:06.512 INFO:teuthology.orchestra.run.smithi079.stderr: 46800K .......... .......... .......... .......... .......... 63% 376M 0s 2019-12-11T16:46:06.512 INFO:teuthology.orchestra.run.smithi079.stderr: 46850K .......... .......... .......... .......... .......... 63% 299M 0s 2019-12-11T16:46:06.512 INFO:teuthology.orchestra.run.smithi079.stderr: 46900K .......... .......... .......... .......... .......... 63% 318M 0s 2019-12-11T16:46:06.513 INFO:teuthology.orchestra.run.smithi079.stderr: 46950K .......... .......... .......... .......... .......... 63% 274M 0s 2019-12-11T16:46:06.513 INFO:teuthology.orchestra.run.smithi079.stderr: 47000K .......... .......... .......... .......... .......... 63% 307M 0s 2019-12-11T16:46:06.513 INFO:teuthology.orchestra.run.smithi079.stderr: 47050K .......... .......... .......... .......... .......... 64% 319M 0s 2019-12-11T16:46:06.513 INFO:teuthology.orchestra.run.smithi079.stderr: 47100K .......... .......... .......... .......... .......... 64% 395M 0s 2019-12-11T16:46:06.513 INFO:teuthology.orchestra.run.smithi079.stderr: 47150K .......... .......... .......... .......... .......... 64% 317M 0s 2019-12-11T16:46:06.513 INFO:teuthology.orchestra.run.smithi079.stderr: 47200K .......... .......... .......... .......... .......... 64% 321M 0s 2019-12-11T16:46:06.513 INFO:teuthology.orchestra.run.smithi079.stderr: 47250K .......... .......... .......... .......... .......... 64% 342M 0s 2019-12-11T16:46:06.514 INFO:teuthology.orchestra.run.smithi079.stderr: 47300K .......... .......... .......... .......... .......... 64% 314M 0s 2019-12-11T16:46:06.514 INFO:teuthology.orchestra.run.smithi079.stderr: 47350K .......... .......... .......... .......... .......... 64% 272M 0s 2019-12-11T16:46:06.514 INFO:teuthology.orchestra.run.smithi079.stderr: 47400K .......... .......... .......... .......... .......... 64% 345M 0s 2019-12-11T16:46:06.514 INFO:teuthology.orchestra.run.smithi079.stderr: 47450K .......... .......... .......... .......... .......... 64% 294M 0s 2019-12-11T16:46:06.514 INFO:teuthology.orchestra.run.smithi079.stderr: 47500K .......... .......... .......... .......... .......... 64% 319M 0s 2019-12-11T16:46:06.514 INFO:teuthology.orchestra.run.smithi079.stderr: 47550K .......... .......... .......... .......... .......... 64% 286M 0s 2019-12-11T16:46:06.514 INFO:teuthology.orchestra.run.smithi079.stderr: 47600K .......... .......... .......... .......... .......... 64% 324M 0s 2019-12-11T16:46:06.515 INFO:teuthology.orchestra.run.smithi079.stderr: 47650K .......... .......... .......... .......... .......... 64% 295M 0s 2019-12-11T16:46:06.515 INFO:teuthology.orchestra.run.smithi079.stderr: 47700K .......... .......... .......... .......... .......... 64% 276M 0s 2019-12-11T16:46:06.515 INFO:teuthology.orchestra.run.smithi079.stderr: 47750K .......... .......... .......... .......... .......... 64% 302M 0s 2019-12-11T16:46:06.515 INFO:teuthology.orchestra.run.smithi079.stderr: 47800K .......... .......... .......... .......... .......... 65% 342M 0s 2019-12-11T16:46:06.515 INFO:teuthology.orchestra.run.smithi079.stderr: 47850K .......... .......... .......... .......... .......... 65% 260M 0s 2019-12-11T16:46:06.515 INFO:teuthology.orchestra.run.smithi079.stderr: 47900K .......... .......... .......... .......... .......... 65% 329M 0s 2019-12-11T16:46:06.515 INFO:teuthology.orchestra.run.smithi079.stderr: 47950K .......... .......... .......... .......... .......... 65% 257M 0s 2019-12-11T16:46:06.516 INFO:teuthology.orchestra.run.smithi079.stderr: 48000K .......... .......... .......... .......... .......... 65% 307M 0s 2019-12-11T16:46:06.516 INFO:teuthology.orchestra.run.smithi079.stderr: 48050K .......... .......... .......... .......... .......... 65% 323M 0s 2019-12-11T16:46:06.516 INFO:teuthology.orchestra.run.smithi079.stderr: 48100K .......... .......... .......... .......... .......... 65% 370M 0s 2019-12-11T16:46:06.516 INFO:teuthology.orchestra.run.smithi079.stderr: 48150K .......... .......... .......... .......... .......... 65% 271M 0s 2019-12-11T16:46:06.516 INFO:teuthology.orchestra.run.smithi079.stderr: 48200K .......... .......... .......... .......... .......... 65% 372M 0s 2019-12-11T16:46:06.516 INFO:teuthology.orchestra.run.smithi079.stderr: 48250K .......... .......... .......... .......... .......... 65% 354M 0s 2019-12-11T16:46:06.517 INFO:teuthology.orchestra.run.smithi079.stderr: 48300K .......... .......... .......... .......... .......... 65% 332M 0s 2019-12-11T16:46:06.517 INFO:teuthology.orchestra.run.smithi079.stderr: 48350K .......... .......... .......... .......... .......... 65% 278M 0s 2019-12-11T16:46:06.517 INFO:teuthology.orchestra.run.smithi079.stderr: 48400K .......... .......... .......... .......... .......... 65% 329M 0s 2019-12-11T16:46:06.517 INFO:teuthology.orchestra.run.smithi079.stderr: 48450K .......... .......... .......... .......... .......... 65% 339M 0s 2019-12-11T16:46:06.517 INFO:teuthology.orchestra.run.smithi079.stderr: 48500K .......... .......... .......... .......... .......... 66% 326M 0s 2019-12-11T16:46:06.517 INFO:teuthology.orchestra.run.smithi079.stderr: 48550K .......... .......... .......... .......... .......... 66% 335M 0s 2019-12-11T16:46:06.518 INFO:teuthology.orchestra.run.smithi079.stderr: 48600K .......... .......... .......... .......... .......... 66% 265M 0s 2019-12-11T16:46:06.518 INFO:teuthology.orchestra.run.smithi079.stderr: 48650K .......... .......... .......... .......... .......... 66% 310M 0s 2019-12-11T16:46:06.518 INFO:teuthology.orchestra.run.smithi079.stderr: 48700K .......... .......... .......... .......... .......... 66% 337M 0s 2019-12-11T16:46:06.518 INFO:teuthology.orchestra.run.smithi079.stderr: 48750K .......... .......... .......... .......... .......... 66% 259M 0s 2019-12-11T16:46:06.518 INFO:teuthology.orchestra.run.smithi079.stderr: 48800K .......... .......... .......... .......... .......... 66% 338M 0s 2019-12-11T16:46:06.518 INFO:teuthology.orchestra.run.smithi079.stderr: 48850K .......... .......... .......... .......... .......... 66% 398M 0s 2019-12-11T16:46:06.518 INFO:teuthology.orchestra.run.smithi079.stderr: 48900K .......... .......... .......... .......... .......... 66% 354M 0s 2019-12-11T16:46:06.519 INFO:teuthology.orchestra.run.smithi079.stderr: 48950K .......... .......... .......... .......... .......... 66% 252M 0s 2019-12-11T16:46:06.519 INFO:teuthology.orchestra.run.smithi079.stderr: 49000K .......... .......... .......... .......... .......... 66% 309M 0s 2019-12-11T16:46:06.519 INFO:teuthology.orchestra.run.smithi079.stderr: 49050K .......... .......... .......... .......... .......... 66% 300M 0s 2019-12-11T16:46:06.519 INFO:teuthology.orchestra.run.smithi079.stderr: 49100K .......... .......... .......... .......... .......... 66% 328M 0s 2019-12-11T16:46:06.519 INFO:teuthology.orchestra.run.smithi079.stderr: 49150K .......... .......... .......... .......... .......... 66% 303M 0s 2019-12-11T16:46:06.519 INFO:teuthology.orchestra.run.smithi079.stderr: 49200K .......... .......... .......... .......... .......... 66% 372M 0s 2019-12-11T16:46:06.520 INFO:teuthology.orchestra.run.smithi079.stderr: 49250K .......... .......... .......... .......... .......... 67% 264M 0s 2019-12-11T16:46:06.520 INFO:teuthology.orchestra.run.smithi079.stderr: 49300K .......... .......... .......... .......... .......... 67% 389M 0s 2019-12-11T16:46:06.520 INFO:teuthology.orchestra.run.smithi079.stderr: 49350K .......... .......... .......... .......... .......... 67% 337M 0s 2019-12-11T16:46:06.520 INFO:teuthology.orchestra.run.smithi079.stderr: 49400K .......... .......... .......... .......... .......... 67% 316M 0s 2019-12-11T16:46:06.520 INFO:teuthology.orchestra.run.smithi079.stderr: 49450K .......... .......... .......... .......... .......... 67% 315M 0s 2019-12-11T16:46:06.521 INFO:teuthology.orchestra.run.smithi079.stderr: 49500K .......... .......... .......... .......... .......... 67% 291M 0s 2019-12-11T16:46:06.521 INFO:teuthology.orchestra.run.smithi079.stderr: 49550K .......... .......... .......... .......... .......... 67% 264M 0s 2019-12-11T16:46:06.521 INFO:teuthology.orchestra.run.smithi079.stderr: 49600K .......... .......... .......... .......... .......... 67% 298M 0s 2019-12-11T16:46:06.521 INFO:teuthology.orchestra.run.smithi079.stderr: 49650K .......... .......... .......... .......... .......... 67% 313M 0s 2019-12-11T16:46:06.521 INFO:teuthology.orchestra.run.smithi079.stderr: 49700K .......... .......... .......... .......... .......... 67% 414M 0s 2019-12-11T16:46:06.521 INFO:teuthology.orchestra.run.smithi079.stderr: 49750K .......... .......... .......... .......... .......... 67% 289M 0s 2019-12-11T16:46:06.521 INFO:teuthology.orchestra.run.smithi079.stderr: 49800K .......... .......... .......... .......... .......... 67% 320M 0s 2019-12-11T16:46:06.522 INFO:teuthology.orchestra.run.smithi079.stderr: 49850K .......... .......... .......... .......... .......... 67% 332M 0s 2019-12-11T16:46:06.522 INFO:teuthology.orchestra.run.smithi079.stderr: 49900K .......... .......... .......... .......... .......... 67% 302M 0s 2019-12-11T16:46:06.522 INFO:teuthology.orchestra.run.smithi079.stderr: 49950K .......... .......... .......... .......... .......... 67% 232M 0s 2019-12-11T16:46:06.522 INFO:teuthology.orchestra.run.smithi079.stderr: 50000K .......... .......... .......... .......... .......... 68% 391M 0s 2019-12-11T16:46:06.522 INFO:teuthology.orchestra.run.smithi079.stderr: 50050K .......... .......... .......... .......... .......... 68% 397M 0s 2019-12-11T16:46:06.522 INFO:teuthology.orchestra.run.smithi079.stderr: 50100K .......... .......... .......... .......... .......... 68% 318M 0s 2019-12-11T16:46:06.522 INFO:teuthology.orchestra.run.smithi079.stderr: 50150K .......... .......... .......... .......... .......... 68% 248M 0s 2019-12-11T16:46:06.523 INFO:teuthology.orchestra.run.smithi079.stderr: 50200K .......... .......... .......... .......... .......... 68% 287M 0s 2019-12-11T16:46:06.523 INFO:teuthology.orchestra.run.smithi079.stderr: 50250K .......... .......... .......... .......... .......... 68% 365M 0s 2019-12-11T16:46:06.523 INFO:teuthology.orchestra.run.smithi079.stderr: 50300K .......... .......... .......... .......... .......... 68% 394M 0s 2019-12-11T16:46:06.523 INFO:teuthology.orchestra.run.smithi079.stderr: 50350K .......... .......... .......... .......... .......... 68% 256M 0s 2019-12-11T16:46:06.523 INFO:teuthology.orchestra.run.smithi079.stderr: 50400K .......... .......... .......... .......... .......... 68% 305M 0s 2019-12-11T16:46:06.523 INFO:teuthology.orchestra.run.smithi079.stderr: 50450K .......... .......... .......... .......... .......... 68% 316M 0s 2019-12-11T16:46:06.524 INFO:teuthology.orchestra.run.smithi079.stderr: 50500K .......... .......... .......... .......... .......... 68% 249M 0s 2019-12-11T16:46:06.524 INFO:teuthology.orchestra.run.smithi079.stderr: 50550K .......... .......... .......... .......... .......... 68% 311M 0s 2019-12-11T16:46:06.524 INFO:teuthology.orchestra.run.smithi079.stderr: 50600K .......... .......... .......... .......... .......... 68% 303M 0s 2019-12-11T16:46:06.524 INFO:teuthology.orchestra.run.smithi079.stderr: 50650K .......... .......... .......... .......... .......... 68% 317M 0s 2019-12-11T16:46:06.524 INFO:teuthology.orchestra.run.smithi079.stderr: 50700K .......... .......... .......... .......... .......... 68% 361M 0s 2019-12-11T16:46:06.524 INFO:teuthology.orchestra.run.smithi079.stderr: 50750K .......... .......... .......... .......... .......... 69% 280M 0s 2019-12-11T16:46:06.524 INFO:teuthology.orchestra.run.smithi079.stderr: 50800K .......... .......... .......... .......... .......... 69% 336M 0s 2019-12-11T16:46:06.525 INFO:teuthology.orchestra.run.smithi079.stderr: 50850K .......... .......... .......... .......... .......... 69% 331M 0s 2019-12-11T16:46:06.525 INFO:teuthology.orchestra.run.smithi079.stderr: 50900K .......... .......... .......... .......... .......... 69% 324M 0s 2019-12-11T16:46:06.525 INFO:teuthology.orchestra.run.smithi079.stderr: 50950K .......... .......... .......... .......... .......... 69% 268M 0s 2019-12-11T16:46:06.525 INFO:teuthology.orchestra.run.smithi079.stderr: 51000K .......... .......... .......... .......... .......... 69% 301M 0s 2019-12-11T16:46:06.525 INFO:teuthology.orchestra.run.smithi079.stderr: 51050K .......... .......... .......... .......... .......... 69% 398M 0s 2019-12-11T16:46:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 51100K .......... .......... .......... .......... .......... 69% 345M 0s 2019-12-11T16:46:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 51150K .......... .......... .......... .......... .......... 69% 232M 0s 2019-12-11T16:46:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 51200K .......... .......... .......... .......... .......... 69% 329M 0s 2019-12-11T16:46:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 51250K .......... .......... .......... .......... .......... 69% 395M 0s 2019-12-11T16:46:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 51300K .......... .......... .......... .......... .......... 69% 343M 0s 2019-12-11T16:46:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 51350K .......... .......... .......... .......... .......... 69% 295M 0s 2019-12-11T16:46:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 51400K .......... .......... .......... .......... .......... 69% 299M 0s 2019-12-11T16:46:06.527 INFO:teuthology.orchestra.run.smithi079.stderr: 51450K .......... .......... .......... .......... .......... 70% 359M 0s 2019-12-11T16:46:06.527 INFO:teuthology.orchestra.run.smithi079.stderr: 51500K .......... .......... .......... .......... .......... 70% 328M 0s 2019-12-11T16:46:06.527 INFO:teuthology.orchestra.run.smithi079.stderr: 51550K .......... .......... .......... .......... .......... 70% 257M 0s 2019-12-11T16:46:06.527 INFO:teuthology.orchestra.run.smithi079.stderr: 51600K .......... .......... .......... .......... .......... 70% 389M 0s 2019-12-11T16:46:06.527 INFO:teuthology.orchestra.run.smithi079.stderr: 51650K .......... .......... .......... .......... .......... 70% 333M 0s 2019-12-11T16:46:06.527 INFO:teuthology.orchestra.run.smithi079.stderr: 51700K .......... .......... .......... .......... .......... 70% 306M 0s 2019-12-11T16:46:06.527 INFO:teuthology.orchestra.run.smithi079.stderr: 51750K .......... .......... .......... .......... .......... 70% 288M 0s 2019-12-11T16:46:06.528 INFO:teuthology.orchestra.run.smithi079.stderr: 51800K .......... .......... .......... .......... .......... 70% 344M 0s 2019-12-11T16:46:06.528 INFO:teuthology.orchestra.run.smithi079.stderr: 51850K .......... .......... .......... .......... .......... 70% 335M 0s 2019-12-11T16:46:06.528 INFO:teuthology.orchestra.run.smithi079.stderr: 51900K .......... .......... .......... .......... .......... 70% 319M 0s 2019-12-11T16:46:06.528 INFO:teuthology.orchestra.run.smithi079.stderr: 51950K .......... .......... .......... .......... .......... 70% 270M 0s 2019-12-11T16:46:06.528 INFO:teuthology.orchestra.run.smithi079.stderr: 52000K .......... .......... .......... .......... .......... 70% 396M 0s 2019-12-11T16:46:06.529 INFO:teuthology.orchestra.run.smithi079.stderr: 52050K .......... .......... .......... .......... .......... 70% 319M 0s 2019-12-11T16:46:06.529 INFO:teuthology.orchestra.run.smithi079.stderr: 52100K .......... .......... .......... .......... .......... 70% 311M 0s 2019-12-11T16:46:06.529 INFO:teuthology.orchestra.run.smithi079.stderr: 52150K .......... .......... .......... .......... .......... 70% 287M 0s 2019-12-11T16:46:06.529 INFO:teuthology.orchestra.run.smithi079.stderr: 52200K .......... .......... .......... .......... .......... 71% 302M 0s 2019-12-11T16:46:06.529 INFO:teuthology.orchestra.run.smithi079.stderr: 52250K .......... .......... .......... .......... .......... 71% 353M 0s 2019-12-11T16:46:06.529 INFO:teuthology.orchestra.run.smithi079.stderr: 52300K .......... .......... .......... .......... .......... 71% 400M 0s 2019-12-11T16:46:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 52350K .......... .......... .......... .......... .......... 71% 337M 0s 2019-12-11T16:46:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 52400K .......... .......... .......... .......... .......... 71% 366M 0s 2019-12-11T16:46:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 52450K .......... .......... .......... .......... .......... 71% 296M 0s 2019-12-11T16:46:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 52500K .......... .......... .......... .......... .......... 71% 321M 0s 2019-12-11T16:46:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 52550K .......... .......... .......... .......... .......... 71% 295M 0s 2019-12-11T16:46:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 52600K .......... .......... .......... .......... .......... 71% 373M 0s 2019-12-11T16:46:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 52650K .......... .......... .......... .......... .......... 71% 326M 0s 2019-12-11T16:46:06.531 INFO:teuthology.orchestra.run.smithi079.stderr: 52700K .......... .......... .......... .......... .......... 71% 344M 0s 2019-12-11T16:46:06.531 INFO:teuthology.orchestra.run.smithi079.stderr: 52750K .......... .......... .......... .......... .......... 71% 331M 0s 2019-12-11T16:46:06.531 INFO:teuthology.orchestra.run.smithi079.stderr: 52800K .......... .......... .......... .......... .......... 71% 344M 0s 2019-12-11T16:46:06.531 INFO:teuthology.orchestra.run.smithi079.stderr: 52850K .......... .......... .......... .......... .......... 71% 286M 0s 2019-12-11T16:46:06.531 INFO:teuthology.orchestra.run.smithi079.stderr: 52900K .......... .......... .......... .......... .......... 71% 348M 0s 2019-12-11T16:46:06.531 INFO:teuthology.orchestra.run.smithi079.stderr: 52950K .......... .......... .......... .......... .......... 72% 317M 0s 2019-12-11T16:46:06.532 INFO:teuthology.orchestra.run.smithi079.stderr: 53000K .......... .......... .......... .......... .......... 72% 388M 0s 2019-12-11T16:46:06.532 INFO:teuthology.orchestra.run.smithi079.stderr: 53050K .......... .......... .......... .......... .......... 72% 348M 0s 2019-12-11T16:46:06.532 INFO:teuthology.orchestra.run.smithi079.stderr: 53100K .......... .......... .......... .......... .......... 72% 364M 0s 2019-12-11T16:46:06.532 INFO:teuthology.orchestra.run.smithi079.stderr: 53150K .......... .......... .......... .......... .......... 72% 263M 0s 2019-12-11T16:46:06.532 INFO:teuthology.orchestra.run.smithi079.stderr: 53200K .......... .......... .......... .......... .......... 72% 327M 0s 2019-12-11T16:46:06.532 INFO:teuthology.orchestra.run.smithi079.stderr: 53250K .......... .......... .......... .......... .......... 72% 336M 0s 2019-12-11T16:46:06.532 INFO:teuthology.orchestra.run.smithi079.stderr: 53300K .......... .......... .......... .......... .......... 72% 326M 0s 2019-12-11T16:46:06.533 INFO:teuthology.orchestra.run.smithi079.stderr: 53350K .......... .......... .......... .......... .......... 72% 309M 0s 2019-12-11T16:46:06.533 INFO:teuthology.orchestra.run.smithi079.stderr: 53400K .......... .......... .......... .......... .......... 72% 348M 0s 2019-12-11T16:46:06.533 INFO:teuthology.orchestra.run.smithi079.stderr: 53450K .......... .......... .......... .......... .......... 72% 338M 0s 2019-12-11T16:46:06.533 INFO:teuthology.orchestra.run.smithi079.stderr: 53500K .......... .......... .......... .......... .......... 72% 330M 0s 2019-12-11T16:46:06.533 INFO:teuthology.orchestra.run.smithi079.stderr: 53550K .......... .......... .......... .......... .......... 72% 302M 0s 2019-12-11T16:46:06.533 INFO:teuthology.orchestra.run.smithi079.stderr: 53600K .......... .......... .......... .......... .......... 72% 339M 0s 2019-12-11T16:46:06.533 INFO:teuthology.orchestra.run.smithi079.stderr: 53650K .......... .......... .......... .......... .......... 73% 392M 0s 2019-12-11T16:46:06.534 INFO:teuthology.orchestra.run.smithi079.stderr: 53700K .......... .......... .......... .......... .......... 73% 370M 0s 2019-12-11T16:46:06.534 INFO:teuthology.orchestra.run.smithi079.stderr: 53750K .......... .......... .......... .......... .......... 73% 298M 0s 2019-12-11T16:46:06.534 INFO:teuthology.orchestra.run.smithi079.stderr: 53800K .......... .......... .......... .......... .......... 73% 340M 0s 2019-12-11T16:46:06.534 INFO:teuthology.orchestra.run.smithi079.stderr: 53850K .......... .......... .......... .......... .......... 73% 349M 0s 2019-12-11T16:46:06.534 INFO:teuthology.orchestra.run.smithi079.stderr: 53900K .......... .......... .......... .......... .......... 73% 348M 0s 2019-12-11T16:46:06.534 INFO:teuthology.orchestra.run.smithi079.stderr: 53950K .......... .......... .......... .......... .......... 73% 320M 0s 2019-12-11T16:46:06.534 INFO:teuthology.orchestra.run.smithi079.stderr: 54000K .......... .......... .......... .......... .......... 73% 329M 0s 2019-12-11T16:46:06.535 INFO:teuthology.orchestra.run.smithi079.stderr: 54050K .......... .......... .......... .......... .......... 73% 361M 0s 2019-12-11T16:46:06.535 INFO:teuthology.orchestra.run.smithi079.stderr: 54100K .......... .......... .......... .......... .......... 73% 329M 0s 2019-12-11T16:46:06.535 INFO:teuthology.orchestra.run.smithi079.stderr: 54150K .......... .......... .......... .......... .......... 73% 337M 0s 2019-12-11T16:46:06.535 INFO:teuthology.orchestra.run.smithi079.stderr: 54200K .......... .......... .......... .......... .......... 73% 362M 0s 2019-12-11T16:46:06.535 INFO:teuthology.orchestra.run.smithi079.stderr: 54250K .......... .......... .......... .......... .......... 73% 360M 0s 2019-12-11T16:46:06.535 INFO:teuthology.orchestra.run.smithi079.stderr: 54300K .......... .......... .......... .......... .......... 73% 405M 0s 2019-12-11T16:46:06.535 INFO:teuthology.orchestra.run.smithi079.stderr: 54350K .......... .......... .......... .......... .......... 73% 313M 0s 2019-12-11T16:46:06.536 INFO:teuthology.orchestra.run.smithi079.stderr: 54400K .......... .......... .......... .......... .......... 74% 309M 0s 2019-12-11T16:46:06.536 INFO:teuthology.orchestra.run.smithi079.stderr: 54450K .......... .......... .......... .......... .......... 74% 342M 0s 2019-12-11T16:46:06.536 INFO:teuthology.orchestra.run.smithi079.stderr: 54500K .......... .......... .......... .......... .......... 74% 349M 0s 2019-12-11T16:46:06.536 INFO:teuthology.orchestra.run.smithi079.stderr: 54550K .......... .......... .......... .......... .......... 74% 309M 0s 2019-12-11T16:46:06.536 INFO:teuthology.orchestra.run.smithi079.stderr: 54600K .......... .......... .......... .......... .......... 74% 384M 0s 2019-12-11T16:46:06.536 INFO:teuthology.orchestra.run.smithi079.stderr: 54650K .......... .......... .......... .......... .......... 74% 357M 0s 2019-12-11T16:46:06.537 INFO:teuthology.orchestra.run.smithi079.stderr: 54700K .......... .......... .......... .......... .......... 74% 366M 0s 2019-12-11T16:46:06.537 INFO:teuthology.orchestra.run.smithi079.stderr: 54750K .......... .......... .......... .......... .......... 74% 270M 0s 2019-12-11T16:46:06.537 INFO:teuthology.orchestra.run.smithi079.stderr: 54800K .......... .......... .......... .......... .......... 74% 345M 0s 2019-12-11T16:46:06.537 INFO:teuthology.orchestra.run.smithi079.stderr: 54850K .......... .......... .......... .......... .......... 74% 377M 0s 2019-12-11T16:46:06.537 INFO:teuthology.orchestra.run.smithi079.stderr: 54900K .......... .......... .......... .......... .......... 74% 407M 0s 2019-12-11T16:46:06.537 INFO:teuthology.orchestra.run.smithi079.stderr: 54950K .......... .......... .......... .......... .......... 74% 340M 0s 2019-12-11T16:46:06.537 INFO:teuthology.orchestra.run.smithi079.stderr: 55000K .......... .......... .......... .......... .......... 74% 320M 0s 2019-12-11T16:46:06.538 INFO:teuthology.orchestra.run.smithi079.stderr: 55050K .......... .......... .......... .......... .......... 74% 304M 0s 2019-12-11T16:46:06.538 INFO:teuthology.orchestra.run.smithi079.stderr: 55100K .......... .......... .......... .......... .......... 74% 332M 0s 2019-12-11T16:46:06.538 INFO:teuthology.orchestra.run.smithi079.stderr: 55150K .......... .......... .......... .......... .......... 75% 175M 0s 2019-12-11T16:46:06.538 INFO:teuthology.orchestra.run.smithi079.stderr: 55200K .......... .......... .......... .......... .......... 75% 203M 0s 2019-12-11T16:46:06.538 INFO:teuthology.orchestra.run.smithi079.stderr: 55250K .......... .......... .......... .......... .......... 75% 280M 0s 2019-12-11T16:46:06.538 INFO:teuthology.orchestra.run.smithi079.stderr: 55300K .......... .......... .......... .......... .......... 75% 250M 0s 2019-12-11T16:46:06.538 INFO:teuthology.orchestra.run.smithi079.stderr: 55350K .......... .......... .......... .......... .......... 75% 185M 0s 2019-12-11T16:46:06.539 INFO:teuthology.orchestra.run.smithi079.stderr: 55400K .......... .......... .......... .......... .......... 75% 251M 0s 2019-12-11T16:46:06.539 INFO:teuthology.orchestra.run.smithi079.stderr: 55450K .......... .......... .......... .......... .......... 75% 222M 0s 2019-12-11T16:46:06.539 INFO:teuthology.orchestra.run.smithi079.stderr: 55500K .......... .......... .......... .......... .......... 75% 223M 0s 2019-12-11T16:46:06.539 INFO:teuthology.orchestra.run.smithi079.stderr: 55550K .......... .......... .......... .......... .......... 75% 189M 0s 2019-12-11T16:46:06.539 INFO:teuthology.orchestra.run.smithi079.stderr: 55600K .......... .......... .......... .......... .......... 75% 267M 0s 2019-12-11T16:46:06.539 INFO:teuthology.orchestra.run.smithi079.stderr: 55650K .......... .......... .......... .......... .......... 75% 170M 0s 2019-12-11T16:46:06.539 INFO:teuthology.orchestra.run.smithi079.stderr: 55700K .......... .......... .......... .......... .......... 75% 212M 0s 2019-12-11T16:46:06.540 INFO:teuthology.orchestra.run.smithi079.stderr: 55750K .......... .......... .......... .......... .......... 75% 273M 0s 2019-12-11T16:46:06.540 INFO:teuthology.orchestra.run.smithi079.stderr: 55800K .......... .......... .......... .......... .......... 75% 212M 0s 2019-12-11T16:46:06.540 INFO:teuthology.orchestra.run.smithi079.stderr: 55850K .......... .......... .......... .......... .......... 75% 228M 0s 2019-12-11T16:46:06.540 INFO:teuthology.orchestra.run.smithi079.stderr: 55900K .......... .......... .......... .......... .......... 76% 229M 0s 2019-12-11T16:46:06.540 INFO:teuthology.orchestra.run.smithi079.stderr: 55950K .......... .......... .......... .......... .......... 76% 222M 0s 2019-12-11T16:46:06.540 INFO:teuthology.orchestra.run.smithi079.stderr: 56000K .......... .......... .......... .......... .......... 76% 252M 0s 2019-12-11T16:46:06.541 INFO:teuthology.orchestra.run.smithi079.stderr: 56050K .......... .......... .......... .......... .......... 76% 280M 0s 2019-12-11T16:46:06.541 INFO:teuthology.orchestra.run.smithi079.stderr: 56100K .......... .......... .......... .......... .......... 76% 225M 0s 2019-12-11T16:46:06.541 INFO:teuthology.orchestra.run.smithi079.stderr: 56150K .......... .......... .......... .......... .......... 76% 231M 0s 2019-12-11T16:46:06.541 INFO:teuthology.orchestra.run.smithi079.stderr: 56200K .......... .......... .......... .......... .......... 76% 219M 0s 2019-12-11T16:46:06.541 INFO:teuthology.orchestra.run.smithi079.stderr: 56250K .......... .......... .......... .......... .......... 76% 234M 0s 2019-12-11T16:46:06.541 INFO:teuthology.orchestra.run.smithi079.stderr: 56300K .......... .......... .......... .......... .......... 76% 231M 0s 2019-12-11T16:46:06.542 INFO:teuthology.orchestra.run.smithi079.stderr: 56350K .......... .......... .......... .......... .......... 76% 178M 0s 2019-12-11T16:46:06.542 INFO:teuthology.orchestra.run.smithi079.stderr: 56400K .......... .......... .......... .......... .......... 76% 207M 0s 2019-12-11T16:46:06.542 INFO:teuthology.orchestra.run.smithi079.stderr: 56450K .......... .......... .......... .......... .......... 76% 246M 0s 2019-12-11T16:46:06.542 INFO:teuthology.orchestra.run.smithi079.stderr: 56500K .......... .......... .......... .......... .......... 76% 282M 0s 2019-12-11T16:46:06.542 INFO:teuthology.orchestra.run.smithi079.stderr: 56550K .......... .......... .......... .......... .......... 76% 184M 0s 2019-12-11T16:46:06.542 INFO:teuthology.orchestra.run.smithi079.stderr: 56600K .......... .......... .......... .......... .......... 77% 214M 0s 2019-12-11T16:46:06.543 INFO:teuthology.orchestra.run.smithi079.stderr: 56650K .......... .......... .......... .......... .......... 77% 234M 0s 2019-12-11T16:46:06.543 INFO:teuthology.orchestra.run.smithi079.stderr: 56700K .......... .......... .......... .......... .......... 77% 254M 0s 2019-12-11T16:46:06.543 INFO:teuthology.orchestra.run.smithi079.stderr: 56750K .......... .......... .......... .......... .......... 77% 212M 0s 2019-12-11T16:46:06.543 INFO:teuthology.orchestra.run.smithi079.stderr: 56800K .......... .......... .......... .......... .......... 77% 233M 0s 2019-12-11T16:46:06.543 INFO:teuthology.orchestra.run.smithi079.stderr: 56850K .......... .......... .......... .......... .......... 77% 234M 0s 2019-12-11T16:46:06.543 INFO:teuthology.orchestra.run.smithi079.stderr: 56900K .......... .......... .......... .......... .......... 77% 263M 0s 2019-12-11T16:46:06.543 INFO:teuthology.orchestra.run.smithi079.stderr: 56950K .......... .......... .......... .......... .......... 77% 229M 0s 2019-12-11T16:46:06.544 INFO:teuthology.orchestra.run.smithi079.stderr: 57000K .......... .......... .......... .......... .......... 77% 299M 0s 2019-12-11T16:46:06.544 INFO:teuthology.orchestra.run.smithi079.stderr: 57050K .......... .......... .......... .......... .......... 77% 204M 0s 2019-12-11T16:46:06.544 INFO:teuthology.orchestra.run.smithi079.stderr: 57100K .......... .......... .......... .......... .......... 77% 239M 0s 2019-12-11T16:46:06.544 INFO:teuthology.orchestra.run.smithi079.stderr: 57150K .......... .......... .......... .......... .......... 77% 250M 0s 2019-12-11T16:46:06.544 INFO:teuthology.orchestra.run.smithi079.stderr: 57200K .......... .......... .......... .......... .......... 77% 294M 0s 2019-12-11T16:46:06.544 INFO:teuthology.orchestra.run.smithi079.stderr: 57250K .......... .......... .......... .......... .......... 77% 253M 0s 2019-12-11T16:46:06.544 INFO:teuthology.orchestra.run.smithi079.stderr: 57300K .......... .......... .......... .......... .......... 77% 203M 0s 2019-12-11T16:46:06.545 INFO:teuthology.orchestra.run.smithi079.stderr: 57350K .......... .......... .......... .......... .......... 78% 225M 0s 2019-12-11T16:46:06.545 INFO:teuthology.orchestra.run.smithi079.stderr: 57400K .......... .......... .......... .......... .......... 78% 209M 0s 2019-12-11T16:46:06.545 INFO:teuthology.orchestra.run.smithi079.stderr: 57450K .......... .......... .......... .......... .......... 78% 252M 0s 2019-12-11T16:46:06.545 INFO:teuthology.orchestra.run.smithi079.stderr: 57500K .......... .......... .......... .......... .......... 78% 218M 0s 2019-12-11T16:46:06.545 INFO:teuthology.orchestra.run.smithi079.stderr: 57550K .......... .......... .......... .......... .......... 78% 241M 0s 2019-12-11T16:46:06.545 INFO:teuthology.orchestra.run.smithi079.stderr: 57600K .......... .......... .......... .......... .......... 78% 231M 0s 2019-12-11T16:46:06.545 INFO:teuthology.orchestra.run.smithi079.stderr: 57650K .......... .......... .......... .......... .......... 78% 218M 0s 2019-12-11T16:46:06.546 INFO:teuthology.orchestra.run.smithi079.stderr: 57700K .......... .......... .......... .......... .......... 78% 235M 0s 2019-12-11T16:46:06.546 INFO:teuthology.orchestra.run.smithi079.stderr: 57750K .......... .......... .......... .......... .......... 78% 207M 0s 2019-12-11T16:46:06.546 INFO:teuthology.orchestra.run.smithi079.stderr: 57800K .......... .......... .......... .......... .......... 78% 240M 0s 2019-12-11T16:46:06.546 INFO:teuthology.orchestra.run.smithi079.stderr: 57850K .......... .......... .......... .......... .......... 78% 278M 0s 2019-12-11T16:46:06.546 INFO:teuthology.orchestra.run.smithi079.stderr: 57900K .......... .......... .......... .......... .......... 78% 281M 0s 2019-12-11T16:46:06.546 INFO:teuthology.orchestra.run.smithi079.stderr: 57950K .......... .......... .......... .......... .......... 78% 187M 0s 2019-12-11T16:46:06.546 INFO:teuthology.orchestra.run.smithi079.stderr: 58000K .......... .......... .......... .......... .......... 78% 232M 0s 2019-12-11T16:46:06.547 INFO:teuthology.orchestra.run.smithi079.stderr: 58050K .......... .......... .......... .......... .......... 78% 243M 0s 2019-12-11T16:46:06.547 INFO:teuthology.orchestra.run.smithi079.stderr: 58100K .......... .......... .......... .......... .......... 79% 279M 0s 2019-12-11T16:46:06.547 INFO:teuthology.orchestra.run.smithi079.stderr: 58150K .......... .......... .......... .......... .......... 79% 208M 0s 2019-12-11T16:46:06.547 INFO:teuthology.orchestra.run.smithi079.stderr: 58200K .......... .......... .......... .......... .......... 79% 269M 0s 2019-12-11T16:46:06.547 INFO:teuthology.orchestra.run.smithi079.stderr: 58250K .......... .......... .......... .......... .......... 79% 236M 0s 2019-12-11T16:46:06.547 INFO:teuthology.orchestra.run.smithi079.stderr: 58300K .......... .......... .......... .......... .......... 79% 247M 0s 2019-12-11T16:46:06.548 INFO:teuthology.orchestra.run.smithi079.stderr: 58350K .......... .......... .......... .......... .......... 79% 247M 0s 2019-12-11T16:46:06.548 INFO:teuthology.orchestra.run.smithi079.stderr: 58400K .......... .......... .......... .......... .......... 79% 263M 0s 2019-12-11T16:46:06.548 INFO:teuthology.orchestra.run.smithi079.stderr: 58450K .......... .......... .......... .......... .......... 79% 245M 0s 2019-12-11T16:46:06.548 INFO:teuthology.orchestra.run.smithi079.stderr: 58500K .......... .......... .......... .......... .......... 79% 242M 0s 2019-12-11T16:46:06.548 INFO:teuthology.orchestra.run.smithi079.stderr: 58550K .......... .......... .......... .......... .......... 79% 226M 0s 2019-12-11T16:46:06.548 INFO:teuthology.orchestra.run.smithi079.stderr: 58600K .......... .......... .......... .......... .......... 79% 285M 0s 2019-12-11T16:46:06.548 INFO:teuthology.orchestra.run.smithi079.stderr: 58650K .......... .......... .......... .......... .......... 79% 231M 0s 2019-12-11T16:46:06.549 INFO:teuthology.orchestra.run.smithi079.stderr: 58700K .......... .......... .......... .......... .......... 79% 244M 0s 2019-12-11T16:46:06.549 INFO:teuthology.orchestra.run.smithi079.stderr: 58750K .......... .......... .......... .......... .......... 79% 170M 0s 2019-12-11T16:46:06.549 INFO:teuthology.orchestra.run.smithi079.stderr: 58800K .......... .......... .......... .......... .......... 80% 288M 0s 2019-12-11T16:46:06.549 INFO:teuthology.orchestra.run.smithi079.stderr: 58850K .......... .......... .......... .......... .......... 80% 204M 0s 2019-12-11T16:46:06.549 INFO:teuthology.orchestra.run.smithi079.stderr: 58900K .......... .......... .......... .......... .......... 80% 294M 0s 2019-12-11T16:46:06.549 INFO:teuthology.orchestra.run.smithi079.stderr: 58950K .......... .......... .......... .......... .......... 80% 198M 0s 2019-12-11T16:46:06.549 INFO:teuthology.orchestra.run.smithi079.stderr: 59000K .......... .......... .......... .......... .......... 80% 240M 0s 2019-12-11T16:46:06.550 INFO:teuthology.orchestra.run.smithi079.stderr: 59050K .......... .......... .......... .......... .......... 80% 213M 0s 2019-12-11T16:46:06.550 INFO:teuthology.orchestra.run.smithi079.stderr: 59100K .......... .......... .......... .......... .......... 80% 252M 0s 2019-12-11T16:46:06.550 INFO:teuthology.orchestra.run.smithi079.stderr: 59150K .......... .......... .......... .......... .......... 80% 213M 0s 2019-12-11T16:46:06.550 INFO:teuthology.orchestra.run.smithi079.stderr: 59200K .......... .......... .......... .......... .......... 80% 215M 0s 2019-12-11T16:46:06.550 INFO:teuthology.orchestra.run.smithi079.stderr: 59250K .......... .......... .......... .......... .......... 80% 294M 0s 2019-12-11T16:46:06.550 INFO:teuthology.orchestra.run.smithi079.stderr: 59300K .......... .......... .......... .......... .......... 80% 291M 0s 2019-12-11T16:46:06.551 INFO:teuthology.orchestra.run.smithi079.stderr: 59350K .......... .......... .......... .......... .......... 80% 267M 0s 2019-12-11T16:46:06.551 INFO:teuthology.orchestra.run.smithi079.stderr: 59400K .......... .......... .......... .......... .......... 80% 286M 0s 2019-12-11T16:46:06.551 INFO:teuthology.orchestra.run.smithi079.stderr: 59450K .......... .......... .......... .......... .......... 80% 228M 0s 2019-12-11T16:46:06.551 INFO:teuthology.orchestra.run.smithi079.stderr: 59500K .......... .......... .......... .......... .......... 80% 261M 0s 2019-12-11T16:46:06.551 INFO:teuthology.orchestra.run.smithi079.stderr: 59550K .......... .......... .......... .......... .......... 81% 195M 0s 2019-12-11T16:46:06.551 INFO:teuthology.orchestra.run.smithi079.stderr: 59600K .......... .......... .......... .......... .......... 81% 248M 0s 2019-12-11T16:46:06.551 INFO:teuthology.orchestra.run.smithi079.stderr: 59650K .......... .......... .......... .......... .......... 81% 288M 0s 2019-12-11T16:46:06.552 INFO:teuthology.orchestra.run.smithi079.stderr: 59700K .......... .......... .......... .......... .......... 81% 238M 0s 2019-12-11T16:46:06.552 INFO:teuthology.orchestra.run.smithi079.stderr: 59750K .......... .......... .......... .......... .......... 81% 196M 0s 2019-12-11T16:46:06.552 INFO:teuthology.orchestra.run.smithi079.stderr: 59800K .......... .......... .......... .......... .......... 81% 226M 0s 2019-12-11T16:46:06.552 INFO:teuthology.orchestra.run.smithi079.stderr: 59850K .......... .......... .......... .......... .......... 81% 275M 0s 2019-12-11T16:46:06.552 INFO:teuthology.orchestra.run.smithi079.stderr: 59900K .......... .......... .......... .......... .......... 81% 248M 0s 2019-12-11T16:46:06.552 INFO:teuthology.orchestra.run.smithi079.stderr: 59950K .......... .......... .......... .......... .......... 81% 196M 0s 2019-12-11T16:46:06.552 INFO:teuthology.orchestra.run.smithi079.stderr: 60000K .......... .......... .......... .......... .......... 81% 237M 0s 2019-12-11T16:46:06.553 INFO:teuthology.orchestra.run.smithi079.stderr: 60050K .......... .......... .......... .......... .......... 81% 291M 0s 2019-12-11T16:46:06.553 INFO:teuthology.orchestra.run.smithi079.stderr: 60100K .......... .......... .......... .......... .......... 81% 229M 0s 2019-12-11T16:46:06.553 INFO:teuthology.orchestra.run.smithi079.stderr: 60150K .......... .......... .......... .......... .......... 81% 222M 0s 2019-12-11T16:46:06.553 INFO:teuthology.orchestra.run.smithi079.stderr: 60200K .......... .......... .......... .......... .......... 81% 244M 0s 2019-12-11T16:46:06.553 INFO:teuthology.orchestra.run.smithi079.stderr: 60250K .......... .......... .......... .......... .......... 81% 218M 0s 2019-12-11T16:46:06.553 INFO:teuthology.orchestra.run.smithi079.stderr: 60300K .......... .......... .......... .......... .......... 82% 229M 0s 2019-12-11T16:46:06.553 INFO:teuthology.orchestra.run.smithi079.stderr: 60350K .......... .......... .......... .......... .......... 82% 245M 0s 2019-12-11T16:46:06.554 INFO:teuthology.orchestra.run.smithi079.stderr: 60400K .......... .......... .......... .......... .......... 82% 255M 0s 2019-12-11T16:46:06.554 INFO:teuthology.orchestra.run.smithi079.stderr: 60450K .......... .......... .......... .......... .......... 82% 221M 0s 2019-12-11T16:46:06.554 INFO:teuthology.orchestra.run.smithi079.stderr: 60500K .......... .......... .......... .......... .......... 82% 237M 0s 2019-12-11T16:46:06.554 INFO:teuthology.orchestra.run.smithi079.stderr: 60550K .......... .......... .......... .......... .......... 82% 262M 0s 2019-12-11T16:46:06.554 INFO:teuthology.orchestra.run.smithi079.stderr: 60600K .......... .......... .......... .......... .......... 82% 294M 0s 2019-12-11T16:46:06.554 INFO:teuthology.orchestra.run.smithi079.stderr: 60650K .......... .......... .......... .......... .......... 82% 213M 0s 2019-12-11T16:46:06.555 INFO:teuthology.orchestra.run.smithi079.stderr: 60700K .......... .......... .......... .......... .......... 82% 216M 0s 2019-12-11T16:46:06.555 INFO:teuthology.orchestra.run.smithi079.stderr: 60750K .......... .......... .......... .......... .......... 82% 240M 0s 2019-12-11T16:46:06.555 INFO:teuthology.orchestra.run.smithi079.stderr: 60800K .......... .......... .......... .......... .......... 82% 234M 0s 2019-12-11T16:46:06.555 INFO:teuthology.orchestra.run.smithi079.stderr: 60850K .......... .......... .......... .......... .......... 82% 238M 0s 2019-12-11T16:46:06.555 INFO:teuthology.orchestra.run.smithi079.stderr: 60900K .......... .......... .......... .......... .......... 82% 250M 0s 2019-12-11T16:46:06.555 INFO:teuthology.orchestra.run.smithi079.stderr: 60950K .......... .......... .......... .......... .......... 82% 212M 0s 2019-12-11T16:46:06.555 INFO:teuthology.orchestra.run.smithi079.stderr: 61000K .......... .......... .......... .......... .......... 82% 248M 0s 2019-12-11T16:46:06.556 INFO:teuthology.orchestra.run.smithi079.stderr: 61050K .......... .......... .......... .......... .......... 83% 298M 0s 2019-12-11T16:46:06.556 INFO:teuthology.orchestra.run.smithi079.stderr: 61100K .......... .......... .......... .......... .......... 83% 298M 0s 2019-12-11T16:46:06.556 INFO:teuthology.orchestra.run.smithi079.stderr: 61150K .......... .......... .......... .......... .......... 83% 239M 0s 2019-12-11T16:46:06.556 INFO:teuthology.orchestra.run.smithi079.stderr: 61200K .......... .......... .......... .......... .......... 83% 255M 0s 2019-12-11T16:46:06.556 INFO:teuthology.orchestra.run.smithi079.stderr: 61250K .......... .......... .......... .......... .......... 83% 282M 0s 2019-12-11T16:46:06.556 INFO:teuthology.orchestra.run.smithi079.stderr: 61300K .......... .......... .......... .......... .......... 83% 266M 0s 2019-12-11T16:46:06.556 INFO:teuthology.orchestra.run.smithi079.stderr: 61350K .......... .......... .......... .......... .......... 83% 206M 0s 2019-12-11T16:46:06.557 INFO:teuthology.orchestra.run.smithi079.stderr: 61400K .......... .......... .......... .......... .......... 83% 233M 0s 2019-12-11T16:46:06.557 INFO:teuthology.orchestra.run.smithi079.stderr: 61450K .......... .......... .......... .......... .......... 83% 234M 0s 2019-12-11T16:46:06.557 INFO:teuthology.orchestra.run.smithi079.stderr: 61500K .......... .......... .......... .......... .......... 83% 246M 0s 2019-12-11T16:46:06.557 INFO:teuthology.orchestra.run.smithi079.stderr: 61550K .......... .......... .......... .......... .......... 83% 205M 0s 2019-12-11T16:46:06.557 INFO:teuthology.orchestra.run.smithi079.stderr: 61600K .......... .......... .......... .......... .......... 83% 280M 0s 2019-12-11T16:46:06.557 INFO:teuthology.orchestra.run.smithi079.stderr: 61650K .......... .......... .......... .......... .......... 83% 232M 0s 2019-12-11T16:46:06.557 INFO:teuthology.orchestra.run.smithi079.stderr: 61700K .......... .......... .......... .......... .......... 83% 282M 0s 2019-12-11T16:46:06.558 INFO:teuthology.orchestra.run.smithi079.stderr: 61750K .......... .......... .......... .......... .......... 84% 204M 0s 2019-12-11T16:46:06.558 INFO:teuthology.orchestra.run.smithi079.stderr: 61800K .......... .......... .......... .......... .......... 84% 278M 0s 2019-12-11T16:46:06.558 INFO:teuthology.orchestra.run.smithi079.stderr: 61850K .......... .......... .......... .......... .......... 84% 221M 0s 2019-12-11T16:46:06.558 INFO:teuthology.orchestra.run.smithi079.stderr: 61900K .......... .......... .......... .......... .......... 84% 259M 0s 2019-12-11T16:46:06.558 INFO:teuthology.orchestra.run.smithi079.stderr: 61950K .......... .......... .......... .......... .......... 84% 197M 0s 2019-12-11T16:46:06.558 INFO:teuthology.orchestra.run.smithi079.stderr: 62000K .......... .......... .......... .......... .......... 84% 238M 0s 2019-12-11T16:46:06.558 INFO:teuthology.orchestra.run.smithi079.stderr: 62050K .......... .......... .......... .......... .......... 84% 287M 0s 2019-12-11T16:46:06.559 INFO:teuthology.orchestra.run.smithi079.stderr: 62100K .......... .......... .......... .......... .......... 84% 235M 0s 2019-12-11T16:46:06.559 INFO:teuthology.orchestra.run.smithi079.stderr: 62150K .......... .......... .......... .......... .......... 84% 238M 0s 2019-12-11T16:46:06.559 INFO:teuthology.orchestra.run.smithi079.stderr: 62200K .......... .......... .......... .......... .......... 84% 236M 0s 2019-12-11T16:46:06.559 INFO:teuthology.orchestra.run.smithi079.stderr: 62250K .......... .......... .......... .......... .......... 84% 293M 0s 2019-12-11T16:46:06.559 INFO:teuthology.orchestra.run.smithi079.stderr: 62300K .......... .......... .......... .......... .......... 84% 238M 0s 2019-12-11T16:46:06.559 INFO:teuthology.orchestra.run.smithi079.stderr: 62350K .......... .......... .......... .......... .......... 84% 208M 0s 2019-12-11T16:46:06.560 INFO:teuthology.orchestra.run.smithi079.stderr: 62400K .......... .......... .......... .......... .......... 84% 293M 0s 2019-12-11T16:46:06.560 INFO:teuthology.orchestra.run.smithi079.stderr: 62450K .......... .......... .......... .......... .......... 84% 265M 0s 2019-12-11T16:46:06.560 INFO:teuthology.orchestra.run.smithi079.stderr: 62500K .......... .......... .......... .......... .......... 85% 238M 0s 2019-12-11T16:46:06.560 INFO:teuthology.orchestra.run.smithi079.stderr: 62550K .......... .......... .......... .......... .......... 85% 225M 0s 2019-12-11T16:46:06.560 INFO:teuthology.orchestra.run.smithi079.stderr: 62600K .......... .......... .......... .......... .......... 85% 292M 0s 2019-12-11T16:46:06.560 INFO:teuthology.orchestra.run.smithi079.stderr: 62650K .......... .......... .......... .......... .......... 85% 306M 0s 2019-12-11T16:46:06.560 INFO:teuthology.orchestra.run.smithi079.stderr: 62700K .......... .......... .......... .......... .......... 85% 241M 0s 2019-12-11T16:46:06.561 INFO:teuthology.orchestra.run.smithi079.stderr: 62750K .......... .......... .......... .......... .......... 85% 188M 0s 2019-12-11T16:46:06.561 INFO:teuthology.orchestra.run.smithi079.stderr: 62800K .......... .......... .......... .......... .......... 85% 250M 0s 2019-12-11T16:46:06.561 INFO:teuthology.orchestra.run.smithi079.stderr: 62850K .......... .......... .......... .......... .......... 85% 261M 0s 2019-12-11T16:46:06.561 INFO:teuthology.orchestra.run.smithi079.stderr: 62900K .......... .......... .......... .......... .......... 85% 233M 0s 2019-12-11T16:46:06.561 INFO:teuthology.orchestra.run.smithi079.stderr: 62950K .......... .......... .......... .......... .......... 85% 221M 0s 2019-12-11T16:46:06.561 INFO:teuthology.orchestra.run.smithi079.stderr: 63000K .......... .......... .......... .......... .......... 85% 293M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63050K .......... .......... .......... .......... .......... 85% 218M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63100K .......... .......... .......... .......... .......... 85% 219M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63150K .......... .......... .......... .......... .......... 85% 200M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63200K .......... .......... .......... .......... .......... 85% 301M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63250K .......... .......... .......... .......... .......... 86% 200M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63300K .......... .......... .......... .......... .......... 86% 232M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63350K .......... .......... .......... .......... .......... 86% 215M 0s 2019-12-11T16:46:06.562 INFO:teuthology.orchestra.run.smithi079.stderr: 63400K .......... .......... .......... .......... .......... 86% 233M 0s 2019-12-11T16:46:06.563 INFO:teuthology.orchestra.run.smithi079.stderr: 63450K .......... .......... .......... .......... .......... 86% 287M 0s 2019-12-11T16:46:06.563 INFO:teuthology.orchestra.run.smithi079.stderr: 63500K .......... .......... .......... .......... .......... 86% 254M 0s 2019-12-11T16:46:06.563 INFO:teuthology.orchestra.run.smithi079.stderr: 63550K .......... .......... .......... .......... .......... 86% 194M 0s 2019-12-11T16:46:06.563 INFO:teuthology.orchestra.run.smithi079.stderr: 63600K .......... .......... .......... .......... .......... 86% 290M 0s 2019-12-11T16:46:06.563 INFO:teuthology.orchestra.run.smithi079.stderr: 63650K .......... .......... .......... .......... .......... 86% 238M 0s 2019-12-11T16:46:06.563 INFO:teuthology.orchestra.run.smithi079.stderr: 63700K .......... .......... .......... .......... .......... 86% 234M 0s 2019-12-11T16:46:06.563 INFO:teuthology.orchestra.run.smithi079.stderr: 63750K .......... .......... .......... .......... .......... 86% 265M 0s 2019-12-11T16:46:06.564 INFO:teuthology.orchestra.run.smithi079.stderr: 63800K .......... .......... .......... .......... .......... 86% 260M 0s 2019-12-11T16:46:06.564 INFO:teuthology.orchestra.run.smithi079.stderr: 63850K .......... .......... .......... .......... .......... 86% 238M 0s 2019-12-11T16:46:06.564 INFO:teuthology.orchestra.run.smithi079.stderr: 63900K .......... .......... .......... .......... .......... 86% 270M 0s 2019-12-11T16:46:06.564 INFO:teuthology.orchestra.run.smithi079.stderr: 63950K .......... .......... .......... .......... .......... 87% 195M 0s 2019-12-11T16:46:06.564 INFO:teuthology.orchestra.run.smithi079.stderr: 64000K .......... .......... .......... .......... .......... 87% 225M 0s 2019-12-11T16:46:06.564 INFO:teuthology.orchestra.run.smithi079.stderr: 64050K .......... .......... .......... .......... .......... 87% 257M 0s 2019-12-11T16:46:06.565 INFO:teuthology.orchestra.run.smithi079.stderr: 64100K .......... .......... .......... .......... .......... 87% 285M 0s 2019-12-11T16:46:06.565 INFO:teuthology.orchestra.run.smithi079.stderr: 64150K .......... .......... .......... .......... .......... 87% 194M 0s 2019-12-11T16:46:06.565 INFO:teuthology.orchestra.run.smithi079.stderr: 64200K .......... .......... .......... .......... .......... 87% 299M 0s 2019-12-11T16:46:06.565 INFO:teuthology.orchestra.run.smithi079.stderr: 64250K .......... .......... .......... .......... .......... 87% 304M 0s 2019-12-11T16:46:06.565 INFO:teuthology.orchestra.run.smithi079.stderr: 64300K .......... .......... .......... .......... .......... 87% 284M 0s 2019-12-11T16:46:06.565 INFO:teuthology.orchestra.run.smithi079.stderr: 64350K .......... .......... .......... .......... .......... 87% 188M 0s 2019-12-11T16:46:06.566 INFO:teuthology.orchestra.run.smithi079.stderr: 64400K .......... .......... .......... .......... .......... 87% 290M 0s 2019-12-11T16:46:06.566 INFO:teuthology.orchestra.run.smithi079.stderr: 64450K .......... .......... .......... .......... .......... 87% 204M 0s 2019-12-11T16:46:06.566 INFO:teuthology.orchestra.run.smithi079.stderr: 64500K .......... .......... .......... .......... .......... 87% 270M 0s 2019-12-11T16:46:06.566 INFO:teuthology.orchestra.run.smithi079.stderr: 64550K .......... .......... .......... .......... .......... 87% 202M 0s 2019-12-11T16:46:06.566 INFO:teuthology.orchestra.run.smithi079.stderr: 64600K .......... .......... .......... .......... .......... 87% 263M 0s 2019-12-11T16:46:06.567 INFO:teuthology.orchestra.run.smithi079.stderr: 64650K .......... .......... .......... .......... .......... 87% 245M 0s 2019-12-11T16:46:06.567 INFO:teuthology.orchestra.run.smithi079.stderr: 64700K .......... .......... .......... .......... .......... 88% 218M 0s 2019-12-11T16:46:06.567 INFO:teuthology.orchestra.run.smithi079.stderr: 64750K .......... .......... .......... .......... .......... 88% 230M 0s 2019-12-11T16:46:06.567 INFO:teuthology.orchestra.run.smithi079.stderr: 64800K .......... .......... .......... .......... .......... 88% 217M 0s 2019-12-11T16:46:06.567 INFO:teuthology.orchestra.run.smithi079.stderr: 64850K .......... .......... .......... .......... .......... 88% 250M 0s 2019-12-11T16:46:06.567 INFO:teuthology.orchestra.run.smithi079.stderr: 64900K .......... .......... .......... .......... .......... 88% 209M 0s 2019-12-11T16:46:06.568 INFO:teuthology.orchestra.run.smithi079.stderr: 64950K .......... .......... .......... .......... .......... 88% 211M 0s 2019-12-11T16:46:06.568 INFO:teuthology.orchestra.run.smithi079.stderr: 65000K .......... .......... .......... .......... .......... 88% 271M 0s 2019-12-11T16:46:06.568 INFO:teuthology.orchestra.run.smithi079.stderr: 65050K .......... .......... .......... .......... .......... 88% 302M 0s 2019-12-11T16:46:06.568 INFO:teuthology.orchestra.run.smithi079.stderr: 65100K .......... .......... .......... .......... .......... 88% 224M 0s 2019-12-11T16:46:06.568 INFO:teuthology.orchestra.run.smithi079.stderr: 65150K .......... .......... .......... .......... .......... 88% 180M 0s 2019-12-11T16:46:06.568 INFO:teuthology.orchestra.run.smithi079.stderr: 65200K .......... .......... .......... .......... .......... 88% 282M 0s 2019-12-11T16:46:06.568 INFO:teuthology.orchestra.run.smithi079.stderr: 65250K .......... .......... .......... .......... .......... 88% 251M 0s 2019-12-11T16:46:06.569 INFO:teuthology.orchestra.run.smithi079.stderr: 65300K .......... .......... .......... .......... .......... 88% 285M 0s 2019-12-11T16:46:06.569 INFO:teuthology.orchestra.run.smithi079.stderr: 65350K .......... .......... .......... .......... .......... 88% 209M 0s 2019-12-11T16:46:06.569 INFO:teuthology.orchestra.run.smithi079.stderr: 65400K .......... .......... .......... .......... .......... 88% 268M 0s 2019-12-11T16:46:06.569 INFO:teuthology.orchestra.run.smithi079.stderr: 65450K .......... .......... .......... .......... .......... 89% 220M 0s 2019-12-11T16:46:06.569 INFO:teuthology.orchestra.run.smithi079.stderr: 65500K .......... .......... .......... .......... .......... 89% 272M 0s 2019-12-11T16:46:06.569 INFO:teuthology.orchestra.run.smithi079.stderr: 65550K .......... .......... .......... .......... .......... 89% 196M 0s 2019-12-11T16:46:06.569 INFO:teuthology.orchestra.run.smithi079.stderr: 65600K .......... .......... .......... .......... .......... 89% 250M 0s 2019-12-11T16:46:06.570 INFO:teuthology.orchestra.run.smithi079.stderr: 65650K .......... .......... .......... .......... .......... 89% 220M 0s 2019-12-11T16:46:06.570 INFO:teuthology.orchestra.run.smithi079.stderr: 65700K .......... .......... .......... .......... .......... 89% 284M 0s 2019-12-11T16:46:06.570 INFO:teuthology.orchestra.run.smithi079.stderr: 65750K .......... .......... .......... .......... .......... 89% 189M 0s 2019-12-11T16:46:06.570 INFO:teuthology.orchestra.run.smithi079.stderr: 65800K .......... .......... .......... .......... .......... 89% 228M 0s 2019-12-11T16:46:06.570 INFO:teuthology.orchestra.run.smithi079.stderr: 65850K .......... .......... .......... .......... .......... 89% 211M 0s 2019-12-11T16:46:06.570 INFO:teuthology.orchestra.run.smithi079.stderr: 65900K .......... .......... .......... .......... .......... 89% 266M 0s 2019-12-11T16:46:06.570 INFO:teuthology.orchestra.run.smithi079.stderr: 65950K .......... .......... .......... .......... .......... 89% 206M 0s 2019-12-11T16:46:06.571 INFO:teuthology.orchestra.run.smithi079.stderr: 66000K .......... .......... .......... .......... .......... 89% 283M 0s 2019-12-11T16:46:06.571 INFO:teuthology.orchestra.run.smithi079.stderr: 66050K .......... .......... .......... .......... .......... 89% 283M 0s 2019-12-11T16:46:06.571 INFO:teuthology.orchestra.run.smithi079.stderr: 66100K .......... .......... .......... .......... .......... 89% 248M 0s 2019-12-11T16:46:06.571 INFO:teuthology.orchestra.run.smithi079.stderr: 66150K .......... .......... .......... .......... .......... 89% 190M 0s 2019-12-11T16:46:06.571 INFO:teuthology.orchestra.run.smithi079.stderr: 66200K .......... .......... .......... .......... .......... 90% 232M 0s 2019-12-11T16:46:06.571 INFO:teuthology.orchestra.run.smithi079.stderr: 66250K .......... .......... .......... .......... .......... 90% 206M 0s 2019-12-11T16:46:06.571 INFO:teuthology.orchestra.run.smithi079.stderr: 66300K .......... .......... .......... .......... .......... 90% 222M 0s 2019-12-11T16:46:06.572 INFO:teuthology.orchestra.run.smithi079.stderr: 66350K .......... .......... .......... .......... .......... 90% 196M 0s 2019-12-11T16:46:06.572 INFO:teuthology.orchestra.run.smithi079.stderr: 66400K .......... .......... .......... .......... .......... 90% 264M 0s 2019-12-11T16:46:06.572 INFO:teuthology.orchestra.run.smithi079.stderr: 66450K .......... .......... .......... .......... .......... 90% 245M 0s 2019-12-11T16:46:06.572 INFO:teuthology.orchestra.run.smithi079.stderr: 66500K .......... .......... .......... .......... .......... 90% 239M 0s 2019-12-11T16:46:06.572 INFO:teuthology.orchestra.run.smithi079.stderr: 66550K .......... .......... .......... .......... .......... 90% 234M 0s 2019-12-11T16:46:06.572 INFO:teuthology.orchestra.run.smithi079.stderr: 66600K .......... .......... .......... .......... .......... 90% 221M 0s 2019-12-11T16:46:06.572 INFO:teuthology.orchestra.run.smithi079.stderr: 66650K .......... .......... .......... .......... .......... 90% 251M 0s 2019-12-11T16:46:06.573 INFO:teuthology.orchestra.run.smithi079.stderr: 66700K .......... .......... .......... .......... .......... 90% 228M 0s 2019-12-11T16:46:06.573 INFO:teuthology.orchestra.run.smithi079.stderr: 66750K .......... .......... .......... .......... .......... 90% 173M 0s 2019-12-11T16:46:06.573 INFO:teuthology.orchestra.run.smithi079.stderr: 66800K .......... .......... .......... .......... .......... 90% 293M 0s 2019-12-11T16:46:06.573 INFO:teuthology.orchestra.run.smithi079.stderr: 66850K .......... .......... .......... .......... .......... 90% 254M 0s 2019-12-11T16:46:06.573 INFO:teuthology.orchestra.run.smithi079.stderr: 66900K .......... .......... .......... .......... .......... 91% 188M 0s 2019-12-11T16:46:06.573 INFO:teuthology.orchestra.run.smithi079.stderr: 66950K .......... .......... .......... .......... .......... 91% 266M 0s 2019-12-11T16:46:06.573 INFO:teuthology.orchestra.run.smithi079.stderr: 67000K .......... .......... .......... .......... .......... 91% 216M 0s 2019-12-11T16:46:06.574 INFO:teuthology.orchestra.run.smithi079.stderr: 67050K .......... .......... .......... .......... .......... 91% 234M 0s 2019-12-11T16:46:06.574 INFO:teuthology.orchestra.run.smithi079.stderr: 67100K .......... .......... .......... .......... .......... 91% 292M 0s 2019-12-11T16:46:06.574 INFO:teuthology.orchestra.run.smithi079.stderr: 67150K .......... .......... .......... .......... .......... 91% 210M 0s 2019-12-11T16:46:06.574 INFO:teuthology.orchestra.run.smithi079.stderr: 67200K .......... .......... .......... .......... .......... 91% 230M 0s 2019-12-11T16:46:06.574 INFO:teuthology.orchestra.run.smithi079.stderr: 67250K .......... .......... .......... .......... .......... 91% 200M 0s 2019-12-11T16:46:06.575 INFO:teuthology.orchestra.run.smithi079.stderr: 67300K .......... .......... .......... .......... .......... 91% 279M 0s 2019-12-11T16:46:06.575 INFO:teuthology.orchestra.run.smithi079.stderr: 67350K .......... .......... .......... .......... .......... 91% 213M 0s 2019-12-11T16:46:06.575 INFO:teuthology.orchestra.run.smithi079.stderr: 67400K .......... .......... .......... .......... .......... 91% 245M 0s 2019-12-11T16:46:06.575 INFO:teuthology.orchestra.run.smithi079.stderr: 67450K .......... .......... .......... .......... .......... 91% 266M 0s 2019-12-11T16:46:06.575 INFO:teuthology.orchestra.run.smithi079.stderr: 67500K .......... .......... .......... .......... .......... 91% 253M 0s 2019-12-11T16:46:06.575 INFO:teuthology.orchestra.run.smithi079.stderr: 67550K .......... .......... .......... .......... .......... 91% 193M 0s 2019-12-11T16:46:06.575 INFO:teuthology.orchestra.run.smithi079.stderr: 67600K .......... .......... .......... .......... .......... 91% 257M 0s 2019-12-11T16:46:06.576 INFO:teuthology.orchestra.run.smithi079.stderr: 67650K .......... .......... .......... .......... .......... 92% 274M 0s 2019-12-11T16:46:06.576 INFO:teuthology.orchestra.run.smithi079.stderr: 67700K .......... .......... .......... .......... .......... 92% 223M 0s 2019-12-11T16:46:06.576 INFO:teuthology.orchestra.run.smithi079.stderr: 67750K .......... .......... .......... .......... .......... 92% 222M 0s 2019-12-11T16:46:06.576 INFO:teuthology.orchestra.run.smithi079.stderr: 67800K .......... .......... .......... .......... .......... 92% 268M 0s 2019-12-11T16:46:06.576 INFO:teuthology.orchestra.run.smithi079.stderr: 67850K .......... .......... .......... .......... .......... 92% 288M 0s 2019-12-11T16:46:06.576 INFO:teuthology.orchestra.run.smithi079.stderr: 67900K .......... .......... .......... .......... .......... 92% 262M 0s 2019-12-11T16:46:06.576 INFO:teuthology.orchestra.run.smithi079.stderr: 67950K .......... .......... .......... .......... .......... 92% 194M 0s 2019-12-11T16:46:06.577 INFO:teuthology.orchestra.run.smithi079.stderr: 68000K .......... .......... .......... .......... .......... 92% 253M 0s 2019-12-11T16:46:06.577 INFO:teuthology.orchestra.run.smithi079.stderr: 68050K .......... .......... .......... .......... .......... 92% 287M 0s 2019-12-11T16:46:06.577 INFO:teuthology.orchestra.run.smithi079.stderr: 68100K .......... .......... .......... .......... .......... 92% 284M 0s 2019-12-11T16:46:06.577 INFO:teuthology.orchestra.run.smithi079.stderr: 68150K .......... .......... .......... .......... .......... 92% 210M 0s 2019-12-11T16:46:06.577 INFO:teuthology.orchestra.run.smithi079.stderr: 68200K .......... .......... .......... .......... .......... 92% 248M 0s 2019-12-11T16:46:06.577 INFO:teuthology.orchestra.run.smithi079.stderr: 68250K .......... .......... .......... .......... .......... 92% 214M 0s 2019-12-11T16:46:06.578 INFO:teuthology.orchestra.run.smithi079.stderr: 68300K .......... .......... .......... .......... .......... 92% 246M 0s 2019-12-11T16:46:06.578 INFO:teuthology.orchestra.run.smithi079.stderr: 68350K .......... .......... .......... .......... .......... 92% 190M 0s 2019-12-11T16:46:06.578 INFO:teuthology.orchestra.run.smithi079.stderr: 68400K .......... .......... .......... .......... .......... 93% 254M 0s 2019-12-11T16:46:06.578 INFO:teuthology.orchestra.run.smithi079.stderr: 68450K .......... .......... .......... .......... .......... 93% 250M 0s 2019-12-11T16:46:06.578 INFO:teuthology.orchestra.run.smithi079.stderr: 68500K .......... .......... .......... .......... .......... 93% 240M 0s 2019-12-11T16:46:06.578 INFO:teuthology.orchestra.run.smithi079.stderr: 68550K .......... .......... .......... .......... .......... 93% 265M 0s 2019-12-11T16:46:06.578 INFO:teuthology.orchestra.run.smithi079.stderr: 68600K .......... .......... .......... .......... .......... 93% 285M 0s 2019-12-11T16:46:06.579 INFO:teuthology.orchestra.run.smithi079.stderr: 68650K .......... .......... .......... .......... .......... 93% 248M 0s 2019-12-11T16:46:06.579 INFO:teuthology.orchestra.run.smithi079.stderr: 68700K .......... .......... .......... .......... .......... 93% 212M 0s 2019-12-11T16:46:06.579 INFO:teuthology.orchestra.run.smithi079.stderr: 68750K .......... .......... .......... .......... .......... 93% 211M 0s 2019-12-11T16:46:06.579 INFO:teuthology.orchestra.run.smithi079.stderr: 68800K .......... .......... .......... .......... .......... 93% 270M 0s 2019-12-11T16:46:06.579 INFO:teuthology.orchestra.run.smithi079.stderr: 68850K .......... .......... .......... .......... .......... 93% 227M 0s 2019-12-11T16:46:06.579 INFO:teuthology.orchestra.run.smithi079.stderr: 68900K .......... .......... .......... .......... .......... 93% 281M 0s 2019-12-11T16:46:06.579 INFO:teuthology.orchestra.run.smithi079.stderr: 68950K .......... .......... .......... .......... .......... 93% 254M 0s 2019-12-11T16:46:06.580 INFO:teuthology.orchestra.run.smithi079.stderr: 69000K .......... .......... .......... .......... .......... 93% 220M 0s 2019-12-11T16:46:06.580 INFO:teuthology.orchestra.run.smithi079.stderr: 69050K .......... .......... .......... .......... .......... 93% 205M 0s 2019-12-11T16:46:06.580 INFO:teuthology.orchestra.run.smithi079.stderr: 69100K .......... .......... .......... .......... .......... 94% 257M 0s 2019-12-11T16:46:06.580 INFO:teuthology.orchestra.run.smithi079.stderr: 69150K .......... .......... .......... .......... .......... 94% 216M 0s 2019-12-11T16:46:06.580 INFO:teuthology.orchestra.run.smithi079.stderr: 69200K .......... .......... .......... .......... .......... 94% 226M 0s 2019-12-11T16:46:06.580 INFO:teuthology.orchestra.run.smithi079.stderr: 69250K .......... .......... .......... .......... .......... 94% 240M 0s 2019-12-11T16:46:06.580 INFO:teuthology.orchestra.run.smithi079.stderr: 69300K .......... .......... .......... .......... .......... 94% 292M 0s 2019-12-11T16:46:06.581 INFO:teuthology.orchestra.run.smithi079.stderr: 69350K .......... .......... .......... .......... .......... 94% 237M 0s 2019-12-11T16:46:06.581 INFO:teuthology.orchestra.run.smithi079.stderr: 69400K .......... .......... .......... .......... .......... 94% 256M 0s 2019-12-11T16:46:06.581 INFO:teuthology.orchestra.run.smithi079.stderr: 69450K .......... .......... .......... .......... .......... 94% 269M 0s 2019-12-11T16:46:06.581 INFO:teuthology.orchestra.run.smithi079.stderr: 69500K .......... .......... .......... .......... .......... 94% 232M 0s 2019-12-11T16:46:06.581 INFO:teuthology.orchestra.run.smithi079.stderr: 69550K .......... .......... .......... .......... .......... 94% 213M 0s 2019-12-11T16:46:06.581 INFO:teuthology.orchestra.run.smithi079.stderr: 69600K .......... .......... .......... .......... .......... 94% 274M 0s 2019-12-11T16:46:06.581 INFO:teuthology.orchestra.run.smithi079.stderr: 69650K .......... .......... .......... .......... .......... 94% 251M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 69700K .......... .......... .......... .......... .......... 94% 278M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 69750K .......... .......... .......... .......... .......... 94% 205M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 69800K .......... .......... .......... .......... .......... 94% 240M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 69850K .......... .......... .......... .......... .......... 95% 241M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 69900K .......... .......... .......... .......... .......... 95% 292M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 69950K .......... .......... .......... .......... .......... 95% 214M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 70000K .......... .......... .......... .......... .......... 95% 229M 0s 2019-12-11T16:46:06.582 INFO:teuthology.orchestra.run.smithi079.stderr: 70050K .......... .......... .......... .......... .......... 95% 289M 0s 2019-12-11T16:46:06.583 INFO:teuthology.orchestra.run.smithi079.stderr: 70100K .......... .......... .......... .......... .......... 95% 235M 0s 2019-12-11T16:46:06.583 INFO:teuthology.orchestra.run.smithi079.stderr: 70150K .......... .......... .......... .......... .......... 95% 238M 0s 2019-12-11T16:46:06.583 INFO:teuthology.orchestra.run.smithi079.stderr: 70200K .......... .......... .......... .......... .......... 95% 201M 0s 2019-12-11T16:46:06.583 INFO:teuthology.orchestra.run.smithi079.stderr: 70250K .......... .......... .......... .......... .......... 95% 270M 0s 2019-12-11T16:46:06.583 INFO:teuthology.orchestra.run.smithi079.stderr: 70300K .......... .......... .......... .......... .......... 95% 239M 0s 2019-12-11T16:46:06.583 INFO:teuthology.orchestra.run.smithi079.stderr: 70350K .......... .......... .......... .......... .......... 95% 209M 0s 2019-12-11T16:46:06.583 INFO:teuthology.orchestra.run.smithi079.stderr: 70400K .......... .......... .......... .......... .......... 95% 293M 0s 2019-12-11T16:46:06.584 INFO:teuthology.orchestra.run.smithi079.stderr: 70450K .......... .......... .......... .......... .......... 95% 268M 0s 2019-12-11T16:46:06.584 INFO:teuthology.orchestra.run.smithi079.stderr: 70500K .......... .......... .......... .......... .......... 95% 233M 0s 2019-12-11T16:46:06.584 INFO:teuthology.orchestra.run.smithi079.stderr: 70550K .......... .......... .......... .......... .......... 95% 219M 0s 2019-12-11T16:46:06.584 INFO:teuthology.orchestra.run.smithi079.stderr: 70600K .......... .......... .......... .......... .......... 96% 240M 0s 2019-12-11T16:46:06.584 INFO:teuthology.orchestra.run.smithi079.stderr: 70650K .......... .......... .......... .......... .......... 96% 280M 0s 2019-12-11T16:46:06.584 INFO:teuthology.orchestra.run.smithi079.stderr: 70700K .......... .......... .......... .......... .......... 96% 242M 0s 2019-12-11T16:46:06.584 INFO:teuthology.orchestra.run.smithi079.stderr: 70750K .......... .......... .......... .......... .......... 96% 214M 0s 2019-12-11T16:46:06.585 INFO:teuthology.orchestra.run.smithi079.stderr: 70800K .......... .......... .......... .......... .......... 96% 298M 0s 2019-12-11T16:46:06.585 INFO:teuthology.orchestra.run.smithi079.stderr: 70850K .......... .......... .......... .......... .......... 96% 264M 0s 2019-12-11T16:46:06.585 INFO:teuthology.orchestra.run.smithi079.stderr: 70900K .......... .......... .......... .......... .......... 96% 254M 0s 2019-12-11T16:46:06.585 INFO:teuthology.orchestra.run.smithi079.stderr: 70950K .......... .......... .......... .......... .......... 96% 238M 0s 2019-12-11T16:46:06.585 INFO:teuthology.orchestra.run.smithi079.stderr: 71000K .......... .......... .......... .......... .......... 96% 232M 0s 2019-12-11T16:46:06.585 INFO:teuthology.orchestra.run.smithi079.stderr: 71050K .......... .......... .......... .......... .......... 96% 258M 0s 2019-12-11T16:46:06.586 INFO:teuthology.orchestra.run.smithi079.stderr: 71100K .......... .......... .......... .......... .......... 96% 251M 0s 2019-12-11T16:46:06.586 INFO:teuthology.orchestra.run.smithi079.stderr: 71150K .......... .......... .......... .......... .......... 96% 248M 0s 2019-12-11T16:46:06.586 INFO:teuthology.orchestra.run.smithi079.stderr: 71200K .......... .......... .......... .......... .......... 96% 195M 0s 2019-12-11T16:46:06.586 INFO:teuthology.orchestra.run.smithi079.stderr: 71250K .......... .......... .......... .......... .......... 96% 275M 0s 2019-12-11T16:46:06.586 INFO:teuthology.orchestra.run.smithi079.stderr: 71300K .......... .......... .......... .......... .......... 96% 252M 0s 2019-12-11T16:46:06.586 INFO:teuthology.orchestra.run.smithi079.stderr: 71350K .......... .......... .......... .......... .......... 97% 189M 0s 2019-12-11T16:46:06.586 INFO:teuthology.orchestra.run.smithi079.stderr: 71400K .......... .......... .......... .......... .......... 97% 223M 0s 2019-12-11T16:46:06.587 INFO:teuthology.orchestra.run.smithi079.stderr: 71450K .......... .......... .......... .......... .......... 97% 253M 0s 2019-12-11T16:46:06.587 INFO:teuthology.orchestra.run.smithi079.stderr: 71500K .......... .......... .......... .......... .......... 97% 250M 0s 2019-12-11T16:46:06.587 INFO:teuthology.orchestra.run.smithi079.stderr: 71550K .......... .......... .......... .......... .......... 97% 221M 0s 2019-12-11T16:46:06.587 INFO:teuthology.orchestra.run.smithi079.stderr: 71600K .......... .......... .......... .......... .......... 97% 209M 0s 2019-12-11T16:46:06.587 INFO:teuthology.orchestra.run.smithi079.stderr: 71650K .......... .......... .......... .......... .......... 97% 247M 0s 2019-12-11T16:46:06.587 INFO:teuthology.orchestra.run.smithi079.stderr: 71700K .......... .......... .......... .......... .......... 97% 226M 0s 2019-12-11T16:46:06.587 INFO:teuthology.orchestra.run.smithi079.stderr: 71750K .......... .......... .......... .......... .......... 97% 261M 0s 2019-12-11T16:46:06.588 INFO:teuthology.orchestra.run.smithi079.stderr: 71800K .......... .......... .......... .......... .......... 97% 215M 0s 2019-12-11T16:46:06.588 INFO:teuthology.orchestra.run.smithi079.stderr: 71850K .......... .......... .......... .......... .......... 97% 293M 0s 2019-12-11T16:46:06.588 INFO:teuthology.orchestra.run.smithi079.stderr: 71900K .......... .......... .......... .......... .......... 97% 278M 0s 2019-12-11T16:46:06.588 INFO:teuthology.orchestra.run.smithi079.stderr: 71950K .......... .......... .......... .......... .......... 97% 193M 0s 2019-12-11T16:46:06.588 INFO:teuthology.orchestra.run.smithi079.stderr: 72000K .......... .......... .......... .......... .......... 97% 293M 0s 2019-12-11T16:46:06.588 INFO:teuthology.orchestra.run.smithi079.stderr: 72050K .......... .......... .......... .......... .......... 98% 292M 0s 2019-12-11T16:46:06.588 INFO:teuthology.orchestra.run.smithi079.stderr: 72100K .......... .......... .......... .......... .......... 98% 291M 0s 2019-12-11T16:46:06.589 INFO:teuthology.orchestra.run.smithi079.stderr: 72150K .......... .......... .......... .......... .......... 98% 272M 0s 2019-12-11T16:46:06.589 INFO:teuthology.orchestra.run.smithi079.stderr: 72200K .......... .......... .......... .......... .......... 98% 284M 0s 2019-12-11T16:46:06.589 INFO:teuthology.orchestra.run.smithi079.stderr: 72250K .......... .......... .......... .......... .......... 98% 306M 0s 2019-12-11T16:46:06.589 INFO:teuthology.orchestra.run.smithi079.stderr: 72300K .......... .......... .......... .......... .......... 98% 298M 0s 2019-12-11T16:46:06.589 INFO:teuthology.orchestra.run.smithi079.stderr: 72350K .......... .......... .......... .......... .......... 98% 248M 0s 2019-12-11T16:46:06.589 INFO:teuthology.orchestra.run.smithi079.stderr: 72400K .......... .......... .......... .......... .......... 98% 301M 0s 2019-12-11T16:46:06.589 INFO:teuthology.orchestra.run.smithi079.stderr: 72450K .......... .......... .......... .......... .......... 98% 288M 0s 2019-12-11T16:46:06.590 INFO:teuthology.orchestra.run.smithi079.stderr: 72500K .......... .......... .......... .......... .......... 98% 291M 0s 2019-12-11T16:46:06.590 INFO:teuthology.orchestra.run.smithi079.stderr: 72550K .......... .......... .......... .......... .......... 98% 267M 0s 2019-12-11T16:46:06.590 INFO:teuthology.orchestra.run.smithi079.stderr: 72600K .......... .......... .......... .......... .......... 98% 299M 0s 2019-12-11T16:46:06.590 INFO:teuthology.orchestra.run.smithi079.stderr: 72650K .......... .......... .......... .......... .......... 98% 300M 0s 2019-12-11T16:46:06.590 INFO:teuthology.orchestra.run.smithi079.stderr: 72700K .......... .......... .......... .......... .......... 98% 293M 0s 2019-12-11T16:46:06.590 INFO:teuthology.orchestra.run.smithi079.stderr: 72750K .......... .......... .......... .......... .......... 98% 249M 0s 2019-12-11T16:46:06.591 INFO:teuthology.orchestra.run.smithi079.stderr: 72800K .......... .......... .......... .......... .......... 99% 276M 0s 2019-12-11T16:46:06.591 INFO:teuthology.orchestra.run.smithi079.stderr: 72850K .......... .......... .......... .......... .......... 99% 283M 0s 2019-12-11T16:46:06.591 INFO:teuthology.orchestra.run.smithi079.stderr: 72900K .......... .......... .......... .......... .......... 99% 287M 0s 2019-12-11T16:46:06.591 INFO:teuthology.orchestra.run.smithi079.stderr: 72950K .......... .......... .......... .......... .......... 99% 256M 0s 2019-12-11T16:46:06.591 INFO:teuthology.orchestra.run.smithi079.stderr: 73000K .......... .......... .......... .......... .......... 99% 280M 0s 2019-12-11T16:46:06.591 INFO:teuthology.orchestra.run.smithi079.stderr: 73050K .......... .......... .......... .......... .......... 99% 300M 0s 2019-12-11T16:46:06.591 INFO:teuthology.orchestra.run.smithi079.stderr: 73100K .......... .......... .......... .......... .......... 99% 277M 0s 2019-12-11T16:46:06.592 INFO:teuthology.orchestra.run.smithi079.stderr: 73150K .......... .......... .......... .......... .......... 99% 244M 0s 2019-12-11T16:46:06.592 INFO:teuthology.orchestra.run.smithi079.stderr: 73200K .......... .......... .......... .......... .......... 99% 292M 0s 2019-12-11T16:46:06.592 INFO:teuthology.orchestra.run.smithi079.stderr: 73250K .......... .......... .......... .......... .......... 99% 297M 0s 2019-12-11T16:46:06.592 INFO:teuthology.orchestra.run.smithi079.stderr: 73300K .......... .......... .......... .......... .......... 99% 277M 0s 2019-12-11T16:46:06.592 INFO:teuthology.orchestra.run.smithi079.stderr: 73350K .......... .......... .......... .......... .......... 99% 262M 0s 2019-12-11T16:46:06.592 INFO:teuthology.orchestra.run.smithi079.stderr: 73400K .......... .......... .......... .......... .......... 99% 291M 0s 2019-12-11T16:46:06.592 INFO:teuthology.orchestra.run.smithi079.stderr: 73450K .......... .......... .......... .......... .......... 99% 295M 0s 2019-12-11T16:46:06.593 INFO:teuthology.orchestra.run.smithi079.stderr: 73500K .......... .......... .......... .......... .......... 99% 303M 0s 2019-12-11T16:46:06.593 INFO:teuthology.orchestra.run.smithi079.stderr: 73550K ........ 100% 228M=0.3s 2019-12-11T16:46:06.593 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T16:46:06.593 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11 16:46:05 (211 MB/s) - ‘gradle-4.7-bin.zip’ saved [75323957/75323957] 2019-12-11T16:46:06.593 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T16:46:06.593 INFO:teuthology.orchestra.run.smithi079.stdout:Archive: gradle-4.7-bin.zip 2019-12-11T16:46:06.594 INFO:teuthology.orchestra.run.smithi079.stdout: creating: /opt/gradle/gradle-4.7/ 2019-12-11T16:46:06.594 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/getting-started.html 2019-12-11T16:46:06.594 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/LICENSE 2019-12-11T16:46:06.594 INFO:teuthology.orchestra.run.smithi079.stdout: creating: /opt/gradle/gradle-4.7/media/ 2019-12-11T16:46:06.594 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-64x64.png 2019-12-11T16:46:06.594 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-48x48.png 2019-12-11T16:46:06.595 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle.icns 2019-12-11T16:46:06.595 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-24x24.png 2019-12-11T16:46:06.595 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-256x256.png 2019-12-11T16:46:06.595 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-512x512.png 2019-12-11T16:46:06.595 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-16x16.png 2019-12-11T16:46:06.595 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-32x32.png 2019-12-11T16:46:06.596 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-128x128.png 2019-12-11T16:46:06.596 INFO:teuthology.orchestra.run.smithi079.stdout: creating: /opt/gradle/gradle-4.7/init.d/ 2019-12-11T16:46:06.596 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/init.d/readme.txt 2019-12-11T16:46:06.596 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/NOTICE 2019-12-11T16:46:06.596 INFO:teuthology.orchestra.run.smithi079.stdout: creating: /opt/gradle/gradle-4.7/bin/ 2019-12-11T16:46:06.596 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/bin/gradle.bat 2019-12-11T16:46:06.597 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/bin/gradle 2019-12-11T16:46:06.597 INFO:teuthology.orchestra.run.smithi079.stdout: creating: /opt/gradle/gradle-4.7/lib/ 2019-12-11T16:46:06.597 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/annotations-13.0.jar 2019-12-11T16:46:06.597 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-kotlin-dsl-0.16.3.jar 2019-12-11T16:46:06.597 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-kotlin-dsl-tooling-builders-0.16.3.jar 2019-12-11T16:46:06.597 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-kotlin-dsl-tooling-models-0.16.3.jar 2019-12-11T16:46:06.598 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-compiler-embeddable-1.2.31.jar 2019-12-11T16:46:06.598 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-reflect-1.2.31.jar 2019-12-11T16:46:06.598 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-sam-with-receiver-compiler-plugin-1.2.31.jar 2019-12-11T16:46:06.598 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-script-runtime-1.2.31.jar 2019-12-11T16:46:06.598 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-stdlib-1.2.31.jar 2019-12-11T16:46:06.598 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-stdlib-jdk7-1.2.31.jar 2019-12-11T16:46:06.598 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-stdlib-jdk8-1.2.31.jar 2019-12-11T16:46:06.599 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-launcher-4.7.jar 2019-12-11T16:46:06.599 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-runtime-api-info-4.7.jar 2019-12-11T16:46:06.599 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-tooling-api-4.7.jar 2019-12-11T16:46:06.599 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-wrapper-4.7.jar 2019-12-11T16:46:06.599 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-installation-beacon-4.7.jar 2019-12-11T16:46:06.599 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-core-4.7.jar 2019-12-11T16:46:06.599 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-jvm-services-4.7.jar 2019-12-11T16:46:06.600 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-core-api-4.7.jar 2019-12-11T16:46:06.600 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-build-cache-4.7.jar 2019-12-11T16:46:06.600 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-logging-4.7.jar 2019-12-11T16:46:06.600 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-process-services-4.7.jar 2019-12-11T16:46:06.600 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-resources-4.7.jar 2019-12-11T16:46:06.600 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-persistent-cache-4.7.jar 2019-12-11T16:46:06.601 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-native-4.7.jar 2019-12-11T16:46:06.601 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-model-groovy-4.7.jar 2019-12-11T16:46:06.601 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-model-core-4.7.jar 2019-12-11T16:46:06.601 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-base-services-groovy-4.7.jar 2019-12-11T16:46:06.601 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-messaging-4.7.jar 2019-12-11T16:46:06.602 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-base-services-4.7.jar 2019-12-11T16:46:06.602 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-build-option-4.7.jar 2019-12-11T16:46:06.602 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-cli-4.7.jar 2019-12-11T16:46:06.602 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-docs-4.7.jar 2019-12-11T16:46:06.602 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-commons-6.1.1.jar 2019-12-11T16:46:06.602 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-analysis-6.1.1.jar 2019-12-11T16:46:06.602 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-tree-6.1.1.jar 2019-12-11T16:46:06.603 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/kryo-2.20.jar 2019-12-11T16:46:06.603 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/reflectasm-1.07-shaded.jar 2019-12-11T16:46:06.603 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-6.1.1.jar 2019-12-11T16:46:06.603 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-io-2.4.jar 2019-12-11T16:46:06.603 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/jcl-over-slf4j-1.7.16.jar 2019-12-11T16:46:06.604 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/jul-to-slf4j-1.7.16.jar 2019-12-11T16:46:06.604 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/log4j-over-slf4j-1.7.16.jar 2019-12-11T16:46:06.604 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/slf4j-api-1.7.16.jar 2019-12-11T16:46:06.604 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/guava-jdk5-17.0.jar 2019-12-11T16:46:06.604 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/jsr305-2.0.1.jar 2019-12-11T16:46:06.605 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/fastutil-7.2.1-min.jar 2019-12-11T16:46:06.605 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-lang-2.6.jar 2019-12-11T16:46:06.605 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/jcip-annotations-1.0.jar 2019-12-11T16:46:06.605 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/groovy-all-2.4.12.jar 2019-12-11T16:46:06.605 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/ant-1.9.9.jar 2019-12-11T16:46:06.605 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/javax.inject-1.jar 2019-12-11T16:46:06.606 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-collections-3.2.2.jar 2019-12-11T16:46:06.606 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-0.14.jar 2019-12-11T16:46:06.606 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-compress-1.14.jar 2019-12-11T16:46:06.606 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/xml-apis-1.4.01.jar 2019-12-11T16:46:06.606 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/jansi-1.14.jar 2019-12-11T16:46:06.606 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/ant-launcher-1.9.9.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-amd64-libcpp-0.14.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-amd64-libstdcpp-0.14.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-i386-libcpp-0.14.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-i386-libstdcpp-0.14.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-amd64-0.14.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-i386-0.14.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-osx-amd64-0.14.jar 2019-12-11T16:46:06.607 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-osx-i386-0.14.jar 2019-12-11T16:46:06.608 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-windows-amd64-0.14.jar 2019-12-11T16:46:06.608 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-windows-i386-0.14.jar 2019-12-11T16:46:06.608 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-amd64-ncurses5-0.14.jar 2019-12-11T16:46:06.608 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-amd64-ncurses6-0.14.jar 2019-12-11T16:46:06.608 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-i386-ncurses5-0.14.jar 2019-12-11T16:46:06.608 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-i386-ncurses6-0.14.jar 2019-12-11T16:46:06.609 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/minlog-1.2.jar 2019-12-11T16:46:06.609 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/objenesis-1.2.jar 2019-12-11T16:46:06.609 INFO:teuthology.orchestra.run.smithi079.stdout: creating: /opt/gradle/gradle-4.7/lib/plugins/ 2019-12-11T16:46:06.609 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-announce-4.7.jar 2019-12-11T16:46:06.609 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-antlr-4.7.jar 2019-12-11T16:46:06.609 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-code-quality-4.7.jar 2019-12-11T16:46:06.609 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-osgi-4.7.jar 2019-12-11T16:46:06.610 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-signing-4.7.jar 2019-12-11T16:46:06.610 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ide-native-4.7.jar 2019-12-11T16:46:06.610 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-native-4.7.jar 2019-12-11T16:46:06.610 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-native-4.7.jar 2019-12-11T16:46:06.610 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-plugin-development-4.7.jar 2019-12-11T16:46:06.610 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-maven-4.7.jar 2019-12-11T16:46:06.610 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ide-play-4.7.jar 2019-12-11T16:46:06.611 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ide-4.7.jar 2019-12-11T16:46:06.611 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-scala-4.7.jar 2019-12-11T16:46:06.611 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-build-comparison-4.7.jar 2019-12-11T16:46:06.611 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ear-4.7.jar 2019-12-11T16:46:06.611 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-play-4.7.jar 2019-12-11T16:46:06.611 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-javascript-4.7.jar 2019-12-11T16:46:06.612 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ivy-4.7.jar 2019-12-11T16:46:06.612 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-jacoco-4.7.jar 2019-12-11T16:46:06.612 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-build-init-4.7.jar 2019-12-11T16:46:06.612 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-plugins-4.7.jar 2019-12-11T16:46:06.612 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-tooling-api-builders-4.7.jar 2019-12-11T16:46:06.613 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-test-kit-4.7.jar 2019-12-11T16:46:06.613 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-composite-builds-4.7.jar 2019-12-11T16:46:06.613 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-groovy-4.7.jar 2019-12-11T16:46:06.613 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-scala-4.7.jar 2019-12-11T16:46:06.613 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-junit-platform-4.7.jar 2019-12-11T16:46:06.613 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-jvm-4.7.jar 2019-12-11T16:46:06.614 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-java-4.7.jar 2019-12-11T16:46:06.614 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-jvm-4.7.jar 2019-12-11T16:46:06.614 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-jvm-4.7.jar 2019-12-11T16:46:06.614 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-native-4.7.jar 2019-12-11T16:46:06.614 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-diagnostics-4.7.jar 2019-12-11T16:46:06.614 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-base-4.7.jar 2019-12-11T16:46:06.614 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-reporting-4.7.jar 2019-12-11T16:46:06.615 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-publish-4.7.jar 2019-12-11T16:46:06.615 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-base-4.7.jar 2019-12-11T16:46:06.615 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-plugin-use-4.7.jar 2019-12-11T16:46:06.615 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-s3-4.7.jar 2019-12-11T16:46:06.615 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-gcs-4.7.jar 2019-12-11T16:46:06.615 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-build-cache-http-4.7.jar 2019-12-11T16:46:06.615 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-dependency-management-4.7.jar 2019-12-11T16:46:06.616 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-http-4.7.jar 2019-12-11T16:46:06.616 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-sftp-4.7.jar 2019-12-11T16:46:06.616 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-workers-4.7.jar 2019-12-11T16:46:06.616 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-version-control-4.7.jar 2019-12-11T16:46:06.616 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aws-java-sdk-s3-1.11.267.jar 2019-12-11T16:46:06.616 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aws-java-sdk-kms-1.11.267.jar 2019-12-11T16:46:06.617 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aws-java-sdk-core-1.11.267.jar 2019-12-11T16:46:06.617 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/org.eclipse.jgit-4.5.3.201708160445-r.jar 2019-12-11T16:46:06.617 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-http-2.4.jar 2019-12-11T16:46:06.617 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-http-shared4-2.4.jar 2019-12-11T16:46:06.617 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-api-services-storage-v1-rev116-1.23.0.jar 2019-12-11T16:46:06.617 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-api-client-1.23.0.jar 2019-12-11T16:46:06.618 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-oauth-client-1.23.0.jar 2019-12-11T16:46:06.618 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-http-client-jackson2-1.23.0.jar 2019-12-11T16:46:06.618 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-http-client-1.23.0.jar 2019-12-11T16:46:06.618 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/httpclient-4.5.5.jar 2019-12-11T16:46:06.618 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/commons-codec-1.10.jar 2019-12-11T16:46:06.618 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-container-default-1.5.5.jar 2019-12-11T16:46:06.618 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/xbean-reflect-3.4.jar 2019-12-11T16:46:06.619 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/asm-util-6.1.1.jar 2019-12-11T16:46:06.619 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jatl-0.2.2.jar 2019-12-11T16:46:06.619 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/commons-cli-1.2.jar 2019-12-11T16:46:06.619 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/biz.aQute.bndlib-3.4.0.jar 2019-12-11T16:46:06.619 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/ivy-2.2.0.jar 2019-12-11T16:46:06.619 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-compat-3.0.4.jar 2019-12-11T16:46:06.620 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-core-3.0.4.jar 2019-12-11T16:46:06.620 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/pmaven-common-0.8-20100325.jar 2019-12-11T16:46:06.620 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/pmaven-groovy-0.8-20100325.jar 2019-12-11T16:46:06.620 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-file-2.4.jar 2019-12-11T16:46:06.620 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-connector-wagon-1.13.1.jar 2019-12-11T16:46:06.620 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/bcpg-jdk15on-1.58.jar 2019-12-11T16:46:06.621 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/rhino-1.7R3.jar 2019-12-11T16:46:06.621 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gson-2.7.jar 2019-12-11T16:46:06.621 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/simple-4.1.21.jar 2019-12-11T16:46:06.621 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-plugin-api-3.0.4.jar 2019-12-11T16:46:06.622 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/testng-6.3.1.jar 2019-12-11T16:46:06.622 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/snakeyaml-1.6.jar 2019-12-11T16:46:06.622 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/dd-plist-1.20.jar 2019-12-11T16:46:06.622 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-4.12.jar 2019-12-11T16:46:06.622 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/bsh-2.0b6.jar 2019-12-11T16:46:06.622 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-platform-launcher-1.0.3.jar 2019-12-11T16:46:06.622 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jcifs-1.3.17.jar 2019-12-11T16:46:06.623 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/nekohtml-1.9.20.jar 2019-12-11T16:46:06.623 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/xercesImpl-2.11.0.jar 2019-12-11T16:46:06.623 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jsch-0.1.54.jar 2019-12-11T16:46:06.623 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jmespath-java-1.11.267.jar 2019-12-11T16:46:06.623 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jackson-databind-2.8.11.jar 2019-12-11T16:46:06.623 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jackson-core-2.8.11.jar 2019-12-11T16:46:06.623 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jackson-annotations-2.8.11.jar 2019-12-11T16:46:06.624 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/joda-time-2.8.2.jar 2019-12-11T16:46:06.624 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/bcprov-jdk15on-1.58.jar 2019-12-11T16:46:06.624 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-aether-provider-3.0.4.jar 2019-12-11T16:46:06.624 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-impl-1.13.1.jar 2019-12-11T16:46:06.624 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-spi-1.13.1.jar 2019-12-11T16:46:06.624 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-util-1.13.1.jar 2019-12-11T16:46:06.625 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-api-1.13.1.jar 2019-12-11T16:46:06.625 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-provider-api-2.4.jar 2019-12-11T16:46:06.625 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-classworlds-2.4.jar 2019-12-11T16:46:06.625 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-model-builder-3.0.4.jar 2019-12-11T16:46:06.625 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-model-3.0.4.jar 2019-12-11T16:46:06.625 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-settings-builder-3.0.4.jar 2019-12-11T16:46:06.625 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-artifact-3.0.4.jar 2019-12-11T16:46:06.626 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-settings-3.0.4.jar 2019-12-11T16:46:06.626 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-sec-dispatcher-1.3.jar 2019-12-11T16:46:06.626 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-repository-metadata-3.0.4.jar 2019-12-11T16:46:06.626 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-utils-3.0.8.jar 2019-12-11T16:46:06.626 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/ion-java-1.0.2.jar 2019-12-11T16:46:06.626 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/httpcore-4.4.9.jar 2019-12-11T16:46:06.626 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/hamcrest-core-1.3.jar 2019-12-11T16:46:06.627 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-platform-engine-1.0.3.jar 2019-12-11T16:46:06.627 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-platform-commons-1.0.3.jar 2019-12-11T16:46:06.627 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/apiguardian-api-1.0.0.jar 2019-12-11T16:46:06.627 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jcommander-1.47.jar 2019-12-11T16:46:06.627 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-interpolation-1.14.jar 2019-12-11T16:46:06.627 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-component-annotations-1.5.5.jar 2019-12-11T16:46:06.627 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/opentest4j-1.0.0.jar 2019-12-11T16:46:06.628 INFO:teuthology.orchestra.run.smithi079.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-cipher-1.7.jar 2019-12-11T16:46:06.628 INFO:teuthology.orchestra.run.smithi079.stdout:export PATH=/opt/gradle/gradle-4.7/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin 2019-12-11T16:46:07.162 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout:Welcome to Gradle 4.7! 2019-12-11T16:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout:Here are the highlights of this release: 2019-12-11T16:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: - Incremental annotation processing 2019-12-11T16:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: - JDK 10 support 2019-12-11T16:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: - Grouped non-interactive console logs 2019-12-11T16:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: - Failed tests are re-run first for quicker feedback 2019-12-11T16:46:07.164 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.164 INFO:teuthology.orchestra.run.smithi079.stdout:For more details see https://docs.gradle.org/4.7/release-notes.html 2019-12-11T16:46:07.164 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.547 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.547 INFO:teuthology.orchestra.run.smithi079.stdout:------------------------------------------------------------ 2019-12-11T16:46:07.547 INFO:teuthology.orchestra.run.smithi079.stdout:Gradle 4.7 2019-12-11T16:46:07.547 INFO:teuthology.orchestra.run.smithi079.stdout:------------------------------------------------------------ 2019-12-11T16:46:07.547 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.547 INFO:teuthology.orchestra.run.smithi079.stdout:Build time: 2018-04-18 09:09:12 UTC 2019-12-11T16:46:07.548 INFO:teuthology.orchestra.run.smithi079.stdout:Revision: b9a962bf70638332300e7f810689cb2febbd4a6c 2019-12-11T16:46:07.548 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.548 INFO:teuthology.orchestra.run.smithi079.stdout:Groovy: 2.4.12 2019-12-11T16:46:07.548 INFO:teuthology.orchestra.run.smithi079.stdout:Ant: Apache Ant(TM) version 1.9.9 compiled on February 2 2017 2019-12-11T16:46:07.548 INFO:teuthology.orchestra.run.smithi079.stdout:JVM: 1.8.0_232 (Oracle Corporation 25.232-b09) 2019-12-11T16:46:07.548 INFO:teuthology.orchestra.run.smithi079.stdout:OS: Linux 3.10.0-1062.9.1.el7.x86_64 amd64 2019-12-11T16:46:07.548 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:07.551 DEBUG:tasks.s3tests_java:S3 Tests Java: BEGIN 2019-12-11T16:46:07.551 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-12-11T16:46:07.551 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi079.front.sepia.ceph.com 2019-12-11T16:46:07.551 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2019-12-11T16:46:07.551 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi199.front.sepia.ceph.com 2019-12-11T16:46:07.552 INFO:tasks.s3tests_java:S3 Tests Java: Creating S3 users... 2019-12-11T16:46:07.946 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf is {'bucket_prefix': 'test-', 's3main': {'access_key': None, 's3dir': 'folder', 'endpoint': None, 'display_name': None, 'region': 'mexico', 'port': None, 'access_secret': None, 'SSE': 'AES256', 'is_secure': None, 'email': None, 'dir': '../data'}} 2019-12-11T16:46:07.947 DEBUG:tasks.s3tests_java:S3 Tests Java: Creating user tester.client.0 2019-12-11T16:46:07.947 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf[s3main] is {'access_key': 'RJFPMBFNNVOLZZTNQUHW', 's3dir': 'folder', 'endpoint': 'smithi079.front.sepia.ceph.com:80', 'display_name': 'Ms. tester.client.0', 'user_id': 'tester.client.0', 'region': 'mexico', 'port': 80, 'access_secret': '5ttmRHDiiZLbnNmN6EPwKZCqEPnl8nXDDlMHyqm2IY94aAyEPJuBQA==', 'host': 'smithi079.front.sepia.ceph.com', 'SSE': 'AES256', 'is_secure': False, 'email': 'tester.client.0_test@test.test', 'dir': '../data'} 2019-12-11T16:46:07.947 DEBUG:tasks.s3tests_java:S3 Tests Java: Setion, User = s3main, tester.client.0 2019-12-11T16:46:07.947 INFO:tasks.s3tests_java:['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '-n', 'client.0', 'user', 'create', '--uid', 'tester.client.0', '--display-name', 'Ms. tester.client.0', '--access-key', 'RJFPMBFNNVOLZZTNQUHW', '--secret', '5ttmRHDiiZLbnNmN6EPwKZCqEPnl8nXDDlMHyqm2IY94aAyEPJuBQA==', '--email', 'tester.client.0_test@test.test', '--cluster', 'ceph'] 2019-12-11T16:46:07.948 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid tester.client.0 --display-name 'Ms. tester.client.0' --access-key RJFPMBFNNVOLZZTNQUHW --secret 5ttmRHDiiZLbnNmN6EPwKZCqEPnl8nXDDlMHyqm2IY94aAyEPJuBQA== --email tester.client.0_test@test.test --cluster ceph 2019-12-11T16:46:08.087 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.104+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.126+0000 7f1237412ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:08.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.126+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.137+0000 7f1237412ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:08.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.137+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.141+0000 7f1237412ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:08.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.142+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.132 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.149+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:08.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.347+0000 7f1237412ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:46:08.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.347+0000 7f1237412ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:46:08.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.347+0000 7f1237412ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:46:08.331 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.347+0000 7f1237412ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:46:08.331 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.347+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.337 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.354+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:08.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.354+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.373+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:46:08.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.373+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.383+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:08.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.383+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.391 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.405+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:46:08.391 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.405+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.411+0000 7f1237412ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:08.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.411+0000 7f1237412ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:46:08.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.411+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.406 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.423+0000 7f1237412ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:08.406 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.423+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.414 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.431+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:08.414 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.431+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.432 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.449+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:46:08.432 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.449+0000 7f1237412ec0 20 zonegroup default 2019-12-11T16:46:08.433 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.449+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.437 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.454+0000 7f1237412ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:08.437 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.454+0000 7f1237412ec0 10 Cannot find current period zone using local zone 2019-12-11T16:46:08.437 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.454+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.460+0000 7f1237412ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:08.444 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.460+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.450 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.467+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:08.450 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.467+0000 7f1237412ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:08.464 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.481+0000 7f1237412ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:46:08.464 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.481+0000 7f1237412ec0 20 zone default found 2019-12-11T16:46:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.481+0000 7f1237412ec0 20 started sync module instance, tier type = 2019-12-11T16:46:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.481+0000 7f1237412ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:46:08.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.901+0000 7f1237412ec0 20 add_watcher() i=0 2019-12-11T16:46:08.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.901+0000 7f1237412ec0 20 add_watcher() i=1 2019-12-11T16:46:08.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.901+0000 7f1237412ec0 20 add_watcher() i=2 2019-12-11T16:46:08.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.901+0000 7f1237412ec0 20 add_watcher() i=3 2019-12-11T16:46:08.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.901+0000 7f1237412ec0 20 add_watcher() i=4 2019-12-11T16:46:08.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.901+0000 7f1237412ec0 20 add_watcher() i=5 2019-12-11T16:46:08.886 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.901+0000 7f1237412ec0 20 add_watcher() i=6 2019-12-11T16:46:08.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.937+0000 7f1237412ec0 20 add_watcher() i=7 2019-12-11T16:46:08.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.937+0000 7f1237412ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:46:08.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.938+0000 7f11f4fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:46:08.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.940+0000 7f1224e3d700 0 -- 172.21.15.79:0/443832483 >> v2:172.21.15.79:3300/0 conn(0x5586e774bb80 msgr2=0x5586e774bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T16:46:08.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:08.978+0000 7f1224e3d700 0 -- 172.21.15.79:0/3210418101 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7f1168022480 msgr2=0x7f1168024990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:46:10.309 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.325+0000 7f1166ffd700 20 reqs_thread_entry: start 2019-12-11T16:46:10.309 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.326+0000 7f1164ff9700 20 reqs_thread_entry: start 2019-12-11T16:46:10.309 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.326+0000 7f1237412ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:46:10.310 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.326+0000 7f1152ffd700 20 reqs_thread_entry: start 2019-12-11T16:46:10.310 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.326+0000 7f1237412ec0 20 get_system_obj_state: rctx=0x7fff6db3ead8 obj=default.rgw.meta:users.uid:tester.client.0 state=0x5586e77d0da0 s->prefetch_data=0 2019-12-11T16:46:10.310 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.326+0000 7f1237412ec0 10 cache get: name=default.rgw.meta+users.uid+tester.client.0 : miss 2019-12-11T16:46:10.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.333+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x0 2019-12-11T16:46:10.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.333+0000 7f1237412ec0 10 adding default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2019-12-11T16:46:10.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.333+0000 7f1237412ec0 20 get_system_obj_state: rctx=0x7fff6db3eae8 obj=default.rgw.meta:users.email:tester.client.0_test@test.test state=0x5586e77d2b30 s->prefetch_data=0 2019-12-11T16:46:10.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.333+0000 7f1237412ec0 10 cache get: name=default.rgw.meta+users.email+tester.client.0_test@test.test : miss 2019-12-11T16:46:10.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.338+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x0 2019-12-11T16:46:10.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.338+0000 7f1237412ec0 10 adding default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2019-12-11T16:46:10.322 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.338+0000 7f1237412ec0 20 get_system_obj_state: rctx=0x7fff6db3eae8 obj=default.rgw.meta:users.keys:RJFPMBFNNVOLZZTNQUHW state=0x5586e77d2b30 s->prefetch_data=0 2019-12-11T16:46:10.322 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.338+0000 7f1237412ec0 10 cache get: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW : miss 2019-12-11T16:46:10.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.346+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW info.flags=0x0 2019-12-11T16:46:10.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.346+0000 7f1237412ec0 10 adding default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW to cache LRU end 2019-12-11T16:46:10.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.346+0000 7f1237412ec0 20 get_system_obj_state: rctx=0x7fff6db3e6f8 obj=default.rgw.meta:users.keys:RJFPMBFNNVOLZZTNQUHW state=0x5586e77d2b30 s->prefetch_data=0 2019-12-11T16:46:10.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.346+0000 7f1237412ec0 10 cache get: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW : type miss (requested=0x6, cached=0x0) 2019-12-11T16:46:10.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.352+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW info.flags=0x0 2019-12-11T16:46:10.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.352+0000 7f1237412ec0 10 moving default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW to cache LRU end 2019-12-11T16:46:10.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.352+0000 7f1237412ec0 20 get_system_obj_state: rctx=0x7fff6db3e878 obj=default.rgw.meta:users.keys:RJFPMBFNNVOLZZTNQUHW state=0x5586e77d2b30 s->prefetch_data=0 2019-12-11T16:46:10.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.352+0000 7f1237412ec0 10 cache get: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW : type miss (requested=0x6, cached=0x0) 2019-12-11T16:46:10.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.359+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW info.flags=0x0 2019-12-11T16:46:10.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.359+0000 7f1237412ec0 10 moving default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW to cache LRU end 2019-12-11T16:46:10.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.402+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2019-12-11T16:46:10.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.402+0000 7f1237412ec0 10 moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2019-12-11T16:46:10.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.402+0000 7f1237412ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=632 2019-12-11T16:46:10.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.418+0000 7f1219ffb700 10 RGWWatcher::handle_notify() notify_id 579820584962 cookie 94037897100848 notifier 4481 bl.length()=632 2019-12-11T16:46:10.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.418+0000 7f1219ffb700 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2019-12-11T16:46:10.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.418+0000 7f1219ffb700 10 moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2019-12-11T16:46:10.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.983+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2019-12-11T16:46:10.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.983+0000 7f1237412ec0 10 moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2019-12-11T16:46:10.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:10.983+0000 7f1237412ec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=200 2019-12-11T16:46:10.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.004+0000 7f1219ffb700 10 RGWWatcher::handle_notify() notify_id 579820584963 cookie 94037897082032 notifier 4481 bl.length()=200 2019-12-11T16:46:10.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.004+0000 7f1219ffb700 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2019-12-11T16:46:10.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.004+0000 7f1219ffb700 10 moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2019-12-11T16:46:11.228 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.244+0000 7f1237412ec0 10 cache put: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW info.flags=0x7 2019-12-11T16:46:11.229 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.244+0000 7f1237412ec0 10 moving default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW to cache LRU end 2019-12-11T16:46:11.229 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.245+0000 7f1237412ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=189 2019-12-11T16:46:11.245 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.260+0000 7f1219ffb700 10 RGWWatcher::handle_notify() notify_id 579820584962 cookie 94037897072688 notifier 4481 bl.length()=189 2019-12-11T16:46:11.245 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.260+0000 7f1219ffb700 10 cache put: name=default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW info.flags=0x7 2019-12-11T16:46:11.245 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.260+0000 7f1219ffb700 10 moving default.rgw.meta+users.keys+RJFPMBFNNVOLZZTNQUHW to cache LRU end 2019-12-11T16:46:11.322 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T16:46:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "tester.client.0", 2019-12-11T16:46:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Ms. tester.client.0", 2019-12-11T16:46:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "tester.client.0_test@test.test", 2019-12-11T16:46:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T16:46:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T16:46:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [], 2019-12-11T16:46:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [ 2019-12-11T16:46:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T16:46:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "tester.client.0", 2019-12-11T16:46:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "access_key": "RJFPMBFNNVOLZZTNQUHW", 2019-12-11T16:46:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "5ttmRHDiiZLbnNmN6EPwKZCqEPnl8nXDDlMHyqm2IY94aAyEPJuBQA==" 2019-12-11T16:46:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T16:46:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T16:46:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [], 2019-12-11T16:46:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [], 2019-12-11T16:46:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T16:46:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T16:46:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T16:46:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T16:46:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T16:46:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T16:46:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T16:46:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T16:46:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T16:46:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T16:46:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T16:46:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T16:46:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T16:46:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T16:46:11.327 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T16:46:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:11.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.387+0000 7f1237412ec0 20 remove_watcher() i=0 2019-12-11T16:46:11.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.387+0000 7f1237412ec0 2 removed watcher, disabling cache 2019-12-11T16:46:11.426 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.443+0000 7f1237412ec0 20 remove_watcher() i=1 2019-12-11T16:46:11.464 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.481+0000 7f1237412ec0 20 remove_watcher() i=2 2019-12-11T16:46:11.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.542+0000 7f1237412ec0 20 remove_watcher() i=3 2019-12-11T16:46:11.587 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.604+0000 7f1237412ec0 20 remove_watcher() i=4 2019-12-11T16:46:11.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.662+0000 7f1237412ec0 20 remove_watcher() i=5 2019-12-11T16:46:11.694 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.711+0000 7f1237412ec0 20 remove_watcher() i=6 2019-12-11T16:46:11.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:11.771+0000 7f1237412ec0 20 remove_watcher() i=7 2019-12-11T16:46:11.765 INFO:teuthology.orchestra.run.smithi079:> cat > /home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf 2019-12-11T16:46:11.831 INFO:tasks.s3tests_java:S3 Tests Java: Running tests... 2019-12-11T16:46:11.832 INFO:teuthology.orchestra.run.smithi079:> cp /home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf /home/ubuntu/cephtest/s3-tests-java/config.properties 2019-12-11T16:46:11.908 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T16:46:12.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.081+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.088 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.105+0000 7f5b0d302ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:12.088 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.105+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.102 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.119+0000 7f5b0d302ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:12.103 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.119+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.125+0000 7f5b0d302ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:12.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.125+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.116 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.133+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:12.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.313+0000 7f5b0d302ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:46:12.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.313+0000 7f5b0d302ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:46:12.301 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.313+0000 7f5b0d302ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:46:12.301 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.313+0000 7f5b0d302ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:46:12.301 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.313+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.307 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.324+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:12.307 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.324+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.336+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:46:12.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.336+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.352+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:12.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.352+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.353 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.370+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:46:12.353 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.370+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.378 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.395+0000 7f5b0d302ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:12.378 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.395+0000 7f5b0d302ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:46:12.378 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.395+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.402+0000 7f5b0d302ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:12.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.402+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.412+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:12.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.412+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.426+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:46:12.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.426+0000 7f5b0d302ec0 20 zonegroup default 2019-12-11T16:46:12.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.426+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.432+0000 7f5b0d302ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:12.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.432+0000 7f5b0d302ec0 10 Cannot find current period zone using local zone 2019-12-11T16:46:12.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.432+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.437+0000 7f5b0d302ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:12.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.437+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.445+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:12.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.445+0000 7f5b0d302ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:12.447 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.464+0000 7f5b0d302ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:46:12.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.464+0000 7f5b0d302ec0 20 zone default found 2019-12-11T16:46:12.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.464+0000 7f5b0d302ec0 20 started sync module instance, tier type = 2019-12-11T16:46:12.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.464+0000 7f5b0d302ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:46:12.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.862+0000 7f5b0d302ec0 20 add_watcher() i=0 2019-12-11T16:46:12.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.862+0000 7f5b0d302ec0 20 add_watcher() i=1 2019-12-11T16:46:12.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.862+0000 7f5b0d302ec0 20 add_watcher() i=2 2019-12-11T16:46:12.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.862+0000 7f5b0d302ec0 20 add_watcher() i=3 2019-12-11T16:46:12.846 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.862+0000 7f5b0d302ec0 20 add_watcher() i=4 2019-12-11T16:46:12.846 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.862+0000 7f5b0d302ec0 20 add_watcher() i=5 2019-12-11T16:46:12.858 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.875+0000 7f5b0d302ec0 20 add_watcher() i=6 2019-12-11T16:46:12.898 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.915+0000 7f5b0d302ec0 20 add_watcher() i=7 2019-12-11T16:46:12.898 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.915+0000 7f5b0d302ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:46:12.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.915+0000 7f5acafdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:46:12.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.932+0000 7f5afa52c700 0 -- 172.21.15.79:0/973133483 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x55daa602d130 msgr2=0x55daa602a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:46:12.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:12.967+0000 7f5af3fff700 0 -- 172.21.15.79:0/1394428451 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x55daa60574f0 msgr2=0x55daa6059910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-11T16:46:14.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.507+0000 7f5a397fa700 20 reqs_thread_entry: start 2019-12-11T16:46:14.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.508+0000 7f5a32ffd700 20 reqs_thread_entry: start 2019-12-11T16:46:14.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.508+0000 7f5b0d302ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:46:14.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.508+0000 7f5a317fa700 20 reqs_thread_entry: start 2019-12-11T16:46:14.492 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.508+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T16:46:14.679 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.696+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:14.705 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.722+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:14.738 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T16:46:14.738 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.40', time=2019-12-11T18:45:36.299587+0000, chain.objs.size()=4 2019-12-11T16:46:14.738 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3__multipart_foo.2~PIpvZDQCNvIWtHpld1LDRgPfm48YM6W.1 2019-12-11T16:46:14.738 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3__multipart_foo.2~PIpvZDQCNvIWtHpld1LDRgPfm48YM6W.2 2019-12-11T16:46:14.738 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3__multipart_foo.2~PIpvZDQCNvIWtHpld1LDRgPfm48YM6W.3 2019-12-11T16:46:14.739 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.3__multipart_foo.2~PIpvZDQCNvIWtHpld1LDRgPfm48YM6W.4 2019-12-11T16:46:14.739 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_3tBuxl1qoU23OGFJu0bsB-R5gAI19A0', time=2019-12-11T18:45:45.953367+0000, chain.objs.size()=1 2019-12-11T16:46:14.739 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.754+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.FaWTPjAUJhqPrdYOwerC94ze5zPayRY_0 2019-12-11T16:46:14.864 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.881+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:14.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:14.999+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T16:46:15.021 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.036+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:15.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.051+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:15.042 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.059+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:15.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.088+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T16:46:15.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.124+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:15.121 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.138+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:15.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T16:46:15.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.114', time=2019-12-11T18:45:41.544855+0000, chain.objs.size()=2 2019-12-11T16:46:15.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.kEkiROKgwBgnmkXYaNxEr_smlT6isbm_0 2019-12-11T16:46:15.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.kEkiROKgwBgnmkXYaNxEr_smlT6isbm_1 2019-12-11T16:46:15.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.224', time=2019-12-11T18:45:43.845115+0000, chain.objs.size()=5 2019-12-11T16:46:15.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.1 2019-12-11T16:46:15.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.1_1 2019-12-11T16:46:15.137 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.2 2019-12-11T16:46:15.137 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.2_1 2019-12-11T16:46:15.137 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.3 2019-12-11T16:46:15.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.265+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:15.353 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.370+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T16:46:15.422 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.439+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:15.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.459+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.468+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:15.497 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.514+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T16:46:15.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.589+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:15.601 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.618+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:15.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.663+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:15.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.663+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.84', time=2019-12-11T18:45:48.634414+0000, chain.objs.size()=3 2019-12-11T16:46:15.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.663+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6__shadow_.OuACHTOE_X3c7nKXoCWG59l8cH9ij8__0 2019-12-11T16:46:15.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.663+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6__shadow_.OuACHTOE_X3c7nKXoCWG59l8cH9ij8__1 2019-12-11T16:46:15.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.663+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6__shadow_.OuACHTOE_X3c7nKXoCWG59l8cH9ij8__2 2019-12-11T16:46:15.867 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:15.884+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:16.039 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.056+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T16:46:16.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.137+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:16.144 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.161+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:16.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.175+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:16.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.217+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T16:46:16.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.300+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:16.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.334+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:16.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.354+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:16.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.354+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.210', time=2019-12-11T18:45:47.428315+0000, chain.objs.size()=3 2019-12-11T16:46:16.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.354+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-STANDARD.2~rr49LN5oYSeIipPmkTdFhq7E3N5dj9b.1 2019-12-11T16:46:16.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.354+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-STANDARD.2~rr49LN5oYSeIipPmkTdFhq7E3N5dj9b.2 2019-12-11T16:46:16.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.354+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-STANDARD.2~rr49LN5oYSeIipPmkTdFhq7E3N5dj9b.3 2019-12-11T16:46:16.482 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.499+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:16.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.594+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T16:46:16.620 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.637+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:16.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.653+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:16.671 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2019-12-11T16:46:16.671 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.112', time=2019-12-11T18:45:41.159841+0000, chain.objs.size()=2 2019-12-11T16:46:16.671 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.kEkiROKgwBgnmkXYaNxEr_smlT6isbm_0 2019-12-11T16:46:16.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.kEkiROKgwBgnmkXYaNxEr_smlT6isbm_1 2019-12-11T16:46:16.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_elkAnGy4vt6OWAcRHgAzyhcKRCe06Ij', time=2019-12-11T18:45:44.422836+0000, chain.objs.size()=2 2019-12-11T16:46:16.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.S8C7OW-1lSHCiMJjjfkZnqEwBxEk74Z_1 2019-12-11T16:46:16.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.S8C7OW-1lSHCiMJjjfkZnqEwBxEk74Z_2 2019-12-11T16:46:16.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_epVPX-oNh_mArfKkvH03IkCWvT5ehfS', time=2019-12-11T18:45:44.872054+0000, chain.objs.size()=3 2019-12-11T16:46:16.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.8B0ekE-WGI4lRSqTMkZLX8nw3POTTdJ_0 2019-12-11T16:46:16.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.8B0ekE-WGI4lRSqTMkZLX8nw3POTTdJ_1 2019-12-11T16:46:16.673 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:16.688+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.8B0ekE-WGI4lRSqTMkZLX8nw3POTTdJ_2 2019-12-11T16:46:16.989 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.005+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:17.116 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.133+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T16:46:17.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.181+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:17.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.199+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:17.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.208+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:17.220 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.237+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T16:46:17.297 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.314+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:17.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.352+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:17.382 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.399+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:17.382 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.399+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.242', time=2019-12-11T18:45:45.592802+0000, chain.objs.size()=3 2019-12-11T16:46:17.382 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.399+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-STANDARD.2~GnSCIa18gykIZlvn-dMa394RNhiojgk.1 2019-12-11T16:46:17.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.399+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-STANDARD.2~GnSCIa18gykIZlvn-dMa394RNhiojgk.2 2019-12-11T16:46:17.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.399+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-STANDARD.2~GnSCIa18gykIZlvn-dMa394RNhiojgk.3 2019-12-11T16:46:17.483 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.500+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:17.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.620+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T16:46:17.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.665+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:17.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.681+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:17.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.690+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:17.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.718+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T16:46:17.753 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.770+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:17.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.790+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:17.787 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.801+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:17.787 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.801+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_hU_i6aZMruJxPsmUGMxI2-jdLGVRfvA', time=2019-12-11T18:45:40.716830+0000, chain.objs.size()=2 2019-12-11T16:46:17.787 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.801+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.ICoc9bwxuvlGpfCIDEPCz-Su8hwjgiA_0 2019-12-11T16:46:17.788 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.801+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.ICoc9bwxuvlGpfCIDEPCz-Su8hwjgiA_1 2019-12-11T16:46:17.948 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:17.965+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:18.041 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.058+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T16:46:18.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.117+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.142+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:18.153 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T16:46:18.153 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.55', time=2019-12-11T18:45:37.557809+0000, chain.objs.size()=4 2019-12-11T16:46:18.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4__multipart_foo.2~6nrf8HbiU-nXA5DjPwL2pql-WqPL99I.1 2019-12-11T16:46:18.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4__multipart_foo.2~6nrf8HbiU-nXA5DjPwL2pql-WqPL99I.2 2019-12-11T16:46:18.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4__multipart_foo.2~6nrf8HbiU-nXA5DjPwL2pql-WqPL99I.3 2019-12-11T16:46:18.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.4__multipart_foo.2~6nrf8HbiU-nXA5DjPwL2pql-WqPL99I.4 2019-12-11T16:46:18.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.109', time=2019-12-11T18:45:40.501420+0000, chain.objs.size()=2 2019-12-11T16:46:18.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.pwF8CbFH-xV03sloEcB6H8grDY6pi9B_0 2019-12-11T16:46:18.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.170+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.pwF8CbFH-xV03sloEcB6H8grDY6pi9B_1 2019-12-11T16:46:18.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.326+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:18.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.426+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T16:46:18.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.465+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.464 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.480+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:18.471 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.488+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.501 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.518+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T16:46:18.560 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.577+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.597 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.607+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:18.602 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.619+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.664+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T16:46:18.719 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.736+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.749 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.766+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:18.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.783+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.815 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.832+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T16:46:18.863 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.880+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:18.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.904+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:18.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T16:46:18.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_nwwDKOCnpgi6PvawNQGGb6Qav5S7tEs', time=2019-12-11T18:45:45.774139+0000, chain.objs.size()=1 2019-12-11T16:46:18.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.4XBpgba1LoDFz4k62mLC3YsaoWH0tFn_0 2019-12-11T16:46:18.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.203', time=2019-12-11T18:45:47.298468+0000, chain.objs.size()=5 2019-12-11T16:46:18.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-LUKEWARM.2~IU7_V8MmHTSZZvCIUwoE-epgbQU2fMC.1 2019-12-11T16:46:18.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__shadow_foo-LUKEWARM.2~IU7_V8MmHTSZZvCIUwoE-epgbQU2fMC.1_1 2019-12-11T16:46:18.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-LUKEWARM.2~IU7_V8MmHTSZZvCIUwoE-epgbQU2fMC.2 2019-12-11T16:46:18.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__shadow_foo-LUKEWARM.2~IU7_V8MmHTSZZvCIUwoE-epgbQU2fMC.2_1 2019-12-11T16:46:18.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:18.920+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-LUKEWARM.2~IU7_V8MmHTSZZvCIUwoE-epgbQU2fMC.3 2019-12-11T16:46:19.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.125+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:19.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.213+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T16:46:19.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.280+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.304+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:19.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.329+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:19.313 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.329+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_ReMcfuI01taIs4f1MGQH-6DTLEP-nKz', time=2019-12-11T18:45:40.942789+0000, chain.objs.size()=1 2019-12-11T16:46:19.313 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.329+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.pIEhNjcFe6W_izvo7RaLI-o5IWEzA0a_1 2019-12-11T16:46:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.413+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:19.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.617+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T16:46:19.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.670+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:19.668 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.685+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:19.682 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.699+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:19.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.699+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.225', time=2019-12-11T18:45:44.050172+0000, chain.objs.size()=5 2019-12-11T16:46:19.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.699+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.1 2019-12-11T16:46:19.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.699+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.1_1 2019-12-11T16:46:19.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.699+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.2 2019-12-11T16:46:19.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.699+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.2_1 2019-12-11T16:46:19.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.699+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-FROZEN.2~k5-yP9201CSbrzlJOZxd7dUSSt-Gj6h.3 2019-12-11T16:46:19.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.927+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:19.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:19.999+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T16:46:20.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.052+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:20.052 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.069+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:20.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.101+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:20.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.101+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.235', time=2019-12-11T18:45:45.182776+0000, chain.objs.size()=5 2019-12-11T16:46:20.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.101+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.1 2019-12-11T16:46:20.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.101+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.1_1 2019-12-11T16:46:20.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.101+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.2 2019-12-11T16:46:20.086 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.101+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.2_1 2019-12-11T16:46:20.086 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.101+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.3 2019-12-11T16:46:20.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.181+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:20.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.301+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T16:46:20.327 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.344+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:20.342 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.359+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2019-12-11T16:46:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.245', time=2019-12-11T18:45:46.143793+0000, chain.objs.size()=3 2019-12-11T16:46:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-STANDARD.2~GnSCIa18gykIZlvn-dMa394RNhiojgk.1 2019-12-11T16:46:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-STANDARD.2~GnSCIa18gykIZlvn-dMa394RNhiojgk.2 2019-12-11T16:46:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-STANDARD.2~GnSCIa18gykIZlvn-dMa394RNhiojgk.3 2019-12-11T16:46:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.196', time=2019-12-11T18:45:47.112898+0000, chain.objs.size()=5 2019-12-11T16:46:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-FROZEN.2~FUU3pJruFM1ijlL_mME1U_lBUI6b2Af.1 2019-12-11T16:46:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__shadow_foo-FROZEN.2~FUU3pJruFM1ijlL_mME1U_lBUI6b2Af.1_1 2019-12-11T16:46:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-FROZEN.2~FUU3pJruFM1ijlL_mME1U_lBUI6b2Af.2 2019-12-11T16:46:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__shadow_foo-FROZEN.2~FUU3pJruFM1ijlL_mME1U_lBUI6b2Af.2_1 2019-12-11T16:46:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.8__multipart_foo-FROZEN.2~FUU3pJruFM1ijlL_mME1U_lBUI6b2Af.3 2019-12-11T16:46:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.85', time=2019-12-11T18:45:48.812472+0000, chain.objs.size()=3 2019-12-11T16:46:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6__shadow_.9R1ehgTF6qX9MDz_8LF7werhvymlkAm_0 2019-12-11T16:46:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6__shadow_.9R1ehgTF6qX9MDz_8LF7werhvymlkAm_1 2019-12-11T16:46:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.389+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.6__shadow_.9R1ehgTF6qX9MDz_8LF7werhvymlkAm_2 2019-12-11T16:46:20.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.762+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:20.866 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.883+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T16:46:20.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.937+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:20.939 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.954+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:20.946 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.963+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:20.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:20.998+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T16:46:21.027 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.044+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:21.044 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.060+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:21.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.072+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:21.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.072+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_CqUZHVLLrh2zmwk8_4b8hbt4LAxOW_v', time=2019-12-11T18:45:41.371875+0000, chain.objs.size()=2 2019-12-11T16:46:21.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.072+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.w_7KG8Z1LDK90X41Nm9yMKxDIC6ToWk_0 2019-12-11T16:46:21.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.072+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.w_7KG8Z1LDK90X41Nm9yMKxDIC6ToWk_1 2019-12-11T16:46:21.151 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.169+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:21.230 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.247+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T16:46:21.272 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.289+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:21.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.306+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:21.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.317+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:21.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.317+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_2kF6UOJs0-s2jUp51V2Sdhl8ekQMlOP', time=2019-12-11T18:45:44.235311+0000, chain.objs.size()=3 2019-12-11T16:46:21.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.317+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.Gx-GbXZY6cvn3pu41pkI3UPP69-LQ2v_0 2019-12-11T16:46:21.301 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.317+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.Gx-GbXZY6cvn3pu41pkI3UPP69-LQ2v_1 2019-12-11T16:46:21.301 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.317+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.Gx-GbXZY6cvn3pu41pkI3UPP69-LQ2v_2 2019-12-11T16:46:21.495 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.512+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:21.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.613+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T16:46:21.633 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.650+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:21.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.665+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:21.661 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T16:46:21.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.71', time=2019-12-11T18:45:38.935107+0000, chain.objs.size()=4 2019-12-11T16:46:21.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5__multipart_foo.2~mmiNIKD9EMxB8dlR5TbXIneN0WYzVnu.1 2019-12-11T16:46:21.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5__multipart_foo.2~mmiNIKD9EMxB8dlR5TbXIneN0WYzVnu.2 2019-12-11T16:46:21.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5__multipart_foo.2~mmiNIKD9EMxB8dlR5TbXIneN0WYzVnu.3 2019-12-11T16:46:21.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.5__multipart_foo.2~mmiNIKD9EMxB8dlR5TbXIneN0WYzVnu.4 2019-12-11T16:46:21.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.233', time=2019-12-11T18:45:44.600877+0000, chain.objs.size()=5 2019-12-11T16:46:21.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.1 2019-12-11T16:46:21.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.1_1 2019-12-11T16:46:21.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.2 2019-12-11T16:46:21.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.2_1 2019-12-11T16:46:21.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.678+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__multipart_foo-LUKEWARM.2~xTgyEE_gPWIkWgjmu9WpfczMm70lAjX.3 2019-12-11T16:46:21.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:21.963+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:22.049 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.062+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T16:46:22.098 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.115+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:22.116 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.134+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:22.133 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2019-12-11T16:46:22.133 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.20', time=2019-12-11T18:45:34.857126+0000, chain.objs.size()=4 2019-12-11T16:46:22.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2__multipart_foo.2~_VAskOt6qnHhlp3tnrhMjVRSETOhMFR.1 2019-12-11T16:46:22.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2__multipart_foo.2~_VAskOt6qnHhlp3tnrhMjVRSETOhMFR.2 2019-12-11T16:46:22.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2__multipart_foo.2~_VAskOt6qnHhlp3tnrhMjVRSETOhMFR.3 2019-12-11T16:46:22.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.2__multipart_foo.2~_VAskOt6qnHhlp3tnrhMjVRSETOhMFR.4 2019-12-11T16:46:22.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_8ulQ3CnLOgSdmGeKUAW8-YJ9Am_dmil', time=2019-12-11T18:45:42.161114+0000, chain.objs.size()=1 2019-12-11T16:46:22.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.LqXf0Sb1xpNXDUGvl17wWTOTn3RrogR_0 2019-12-11T16:46:22.135 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_1-85uefwqjc5-0MvB3JMFhwLM4835Km', time=2019-12-11T18:45:42.355169+0000, chain.objs.size()=1 2019-12-11T16:46:22.135 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.150+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.Wq2Uokv_xqS9mIIB_ojUj1BEthqRjYZ_0 2019-12-11T16:46:22.299 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.316+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:22.371 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.388+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T16:46:22.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.433+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:22.432 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.449+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:22.440 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.457+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:22.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.493+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T16:46:22.518 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.536+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:22.535 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.552+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:22.565 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.582+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:22.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.582+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_u5uRLbrOQbKGo6mSzwHIHtfYuwaOwUi', time=2019-12-11T18:45:45.418314+0000, chain.objs.size()=2 2019-12-11T16:46:22.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.582+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.it_PmRzGf8YrDbppCBfOYonPHNJcX0M_1 2019-12-11T16:46:22.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.582+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.9__shadow_.it_PmRzGf8YrDbppCBfOYonPHNJcX0M_2 2019-12-11T16:46:22.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.707+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:22.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.840+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T16:46:22.864 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.881+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:22.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.918+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:22.919 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.936+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:22.919 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.936+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.108', time=2019-12-11T18:45:40.304073+0000, chain.objs.size()=2 2019-12-11T16:46:22.919 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.936+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.pwF8CbFH-xV03sloEcB6H8grDY6pi9B_0 2019-12-11T16:46:22.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:22.936+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.pwF8CbFH-xV03sloEcB6H8grDY6pi9B_1 2019-12-11T16:46:23.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.063+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:23.281 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.298+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T16:46:23.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.337+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:23.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.352+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:23.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.360+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:23.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.391+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T16:46:23.424 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.441+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.454+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:23.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.465+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T16:46:23.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.465+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process iterating over entry tag='_8iZnx_41L4DAYbUNsC3SsWHnqt4ca7H', time=2019-12-11T18:45:41.762930+0000, chain.objs.size()=1 2019-12-11T16:46:23.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.465+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.7__shadow_.kgh49LneTFwzbpmKYNm3VC3bE1WuAHG_1 2019-12-11T16:46:23.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.545+0000 7f5b0d302ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T16:46:23.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.647+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T16:46:23.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.693+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:23.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.707+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:23.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.715+0000 7f5b0d302ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:23.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.826+0000 7f5b0d302ec0 20 remove_watcher() i=0 2019-12-11T16:46:23.812 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.826+0000 7f5b0d302ec0 2 removed watcher, disabling cache 2019-12-11T16:46:23.853 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.868+0000 7f5b0d302ec0 20 remove_watcher() i=1 2019-12-11T16:46:23.897 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.911+0000 7f5b0d302ec0 20 remove_watcher() i=2 2019-12-11T16:46:23.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:23.979+0000 7f5b0d302ec0 20 remove_watcher() i=3 2019-12-11T16:46:24.009 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.021+0000 7f5b0d302ec0 20 remove_watcher() i=4 2019-12-11T16:46:24.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.077+0000 7f5b0d302ec0 20 remove_watcher() i=5 2019-12-11T16:46:24.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.130+0000 7f5b0d302ec0 20 remove_watcher() i=6 2019-12-11T16:46:24.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.178+0000 7f5b0d302ec0 20 remove_watcher() i=7 2019-12-11T16:46:24.171 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T16:46:24.291 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.308+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.322 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.340+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:24.323 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.340+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.333 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.348+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:24.333 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.349+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.353+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:24.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.353+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.361+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:24.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.507+0000 7f6a1c2c4ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T16:46:24.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.507+0000 7f6a1c2c4ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T16:46:24.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.507+0000 7f6a1c2c4ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T16:46:24.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.507+0000 7f6a1c2c4ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T16:46:24.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.507+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.527+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:24.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.527+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.519 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.536+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:46:24.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.536+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.526 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.543+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:24.526 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.543+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.541 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.558+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:46:24.541 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.558+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.562+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:24.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.562+0000 7f6a1c2c4ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T16:46:24.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.562+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.568+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.568+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.559 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.576+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:24.559 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.576+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.573 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.589+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T16:46:24.573 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.589+0000 7f6a1c2c4ec0 20 zonegroup default 2019-12-11T16:46:24.573 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.589+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.594+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:24.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.594+0000 7f6a1c2c4ec0 10 Cannot find current period zone using local zone 2019-12-11T16:46:24.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.594+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.581 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.599+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T16:46:24.582 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.599+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.589 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.606+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T16:46:24.589 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.606+0000 7f6a1c2c4ec0 20 rados->read ofs=0 len=0 2019-12-11T16:46:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.624+0000 7f6a1c2c4ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T16:46:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.624+0000 7f6a1c2c4ec0 20 zone default found 2019-12-11T16:46:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.624+0000 7f6a1c2c4ec0 20 started sync module instance, tier type = 2019-12-11T16:46:24.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.624+0000 7f6a1c2c4ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T16:46:24.975 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.992+0000 7f6a1c2c4ec0 20 add_watcher() i=0 2019-12-11T16:46:24.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.992+0000 7f6a1c2c4ec0 20 add_watcher() i=1 2019-12-11T16:46:24.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.992+0000 7f6a1c2c4ec0 20 add_watcher() i=2 2019-12-11T16:46:24.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.992+0000 7f6a1c2c4ec0 20 add_watcher() i=3 2019-12-11T16:46:24.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.992+0000 7f6a1c2c4ec0 20 add_watcher() i=4 2019-12-11T16:46:24.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.992+0000 7f6a1c2c4ec0 20 add_watcher() i=5 2019-12-11T16:46:24.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:24.992+0000 7f6a1c2c4ec0 20 add_watcher() i=6 2019-12-11T16:46:25.022 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:25.039+0000 7f6a1c2c4ec0 20 add_watcher() i=7 2019-12-11T16:46:25.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:25.039+0000 7f6a1c2c4ec0 2 all 8 watchers are set, enabling cache 2019-12-11T16:46:25.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:25.039+0000 7f69d97e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T16:46:25.027 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:25.044+0000 7f6a08ced700 0 -- 172.21.15.79:0/381136948 >> v2:172.21.15.79:3300/0 conn(0x56269419b4d0 msgr2=0x56269419d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T16:46:25.082 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:25.097+0000 7f6a09cef700 0 -- 172.21.15.79:0/1787222974 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7f694c022490 msgr2=0x7f694c024960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T16:46:26.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.422+0000 7f694b7fe700 20 reqs_thread_entry: start 2019-12-11T16:46:26.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.423+0000 7f69497fa700 20 reqs_thread_entry: start 2019-12-11T16:46:26.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.423+0000 7f6a1c2c4ec0 20 init_complete bucket index max shards: 0 2019-12-11T16:46:26.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.423+0000 7f69377fe700 20 reqs_thread_entry: start 2019-12-11T16:46:26.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.423+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T16:46:26.463 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.480+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.479 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.496+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:26.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.504+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.538 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.554+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T16:46:26.605 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.615+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.619 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.634+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:26.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.645+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.664 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.679+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T16:46:26.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.719+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.721 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.735+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:26.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.744+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.774+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T16:46:26.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.820+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.818 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.835+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:26.826 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.843+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.857 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.874+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T16:46:26.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.953+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:26.969 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:26.986+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:26.993 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.010+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.050 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.066+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T16:46:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.118+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.143+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:27.139 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.154+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.191+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T16:46:27.233 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.250+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.266+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:27.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.275+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.287 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.305+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T16:46:27.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.348+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.353 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.363+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:27.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.374+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.388 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.405+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T16:46:27.437 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.453+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.455 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.472+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:27.468 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.484+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.497 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.514+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T16:46:27.552 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.569+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.570 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.587+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:27.582 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.599+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.657+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T16:46:27.714 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.731+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.752+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:27.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.763+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.795 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.812+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T16:46:27.872 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.889+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.916+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:27.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.930+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:27.968 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:27.984+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T16:46:28.028 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.032+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.051+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.044 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.061+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.077 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.094+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T16:46:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.137+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.153+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.147 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.164+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.202+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T16:46:28.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.255+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.270+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.262 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.279+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.365+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T16:46:28.404 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.421+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.445+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.441 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.458+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.493+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T16:46:28.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.542+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.564+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.557 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.574+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.592 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.609+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T16:46:28.661 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.675+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.705+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.718+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.781+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T16:46:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.853+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.867 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.879+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.873 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.891+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.933+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T16:46:28.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.980+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:28.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:28.996+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:28.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.004+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.018 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.036+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T16:46:29.099 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.116+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.152 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.150+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:29.152 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.169+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.215 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.232+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T16:46:29.290 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.301+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.369+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:29.371 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.388+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.413 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.430+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T16:46:29.465 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.482+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.482 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.500+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:29.492 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.509+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.544+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T16:46:29.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.594+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.601 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.618+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:29.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.633+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.692+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T16:46:29.732 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.749+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.777+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:29.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.792+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:29.834 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:29.851+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T16:46:30.040 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.058+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.062 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.080+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.089+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.130+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T16:46:30.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.200+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.225+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:30.229 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.237+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.271 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.288+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T16:46:30.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.373+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.378 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.392+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:30.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.417+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.463+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T16:46:30.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.538+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.548 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.565+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:30.561 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.579+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.623 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.640+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T16:46:30.671 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.688+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.707+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.719+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.757+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T16:46:30.812 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.830+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.854+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:30.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.869+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.927+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T16:46:30.959 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.977+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:30.997+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T16:46:30.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.009+0000 7f6a1c2c4ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T16:46:31.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.124+0000 7f6a1c2c4ec0 20 remove_watcher() i=0 2019-12-11T16:46:31.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.124+0000 7f6a1c2c4ec0 2 removed watcher, disabling cache 2019-12-11T16:46:31.140 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.157+0000 7f6a1c2c4ec0 20 remove_watcher() i=1 2019-12-11T16:46:31.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.197+0000 7f6a1c2c4ec0 20 remove_watcher() i=2 2019-12-11T16:46:31.237 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.254+0000 7f6a1c2c4ec0 20 remove_watcher() i=3 2019-12-11T16:46:31.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.317+0000 7f6a1c2c4ec0 20 remove_watcher() i=4 2019-12-11T16:46:31.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.368+0000 7f6a1c2c4ec0 20 remove_watcher() i=5 2019-12-11T16:46:31.412 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.430+0000 7f6a1c2c4ec0 20 remove_watcher() i=6 2019-12-11T16:46:31.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T16:46:31.501+0000 7f6a1c2c4ec0 20 remove_watcher() i=7 2019-12-11T16:46:31.495 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test 2019-12-11T16:46:32.132 INFO:teuthology.orchestra.run.smithi079.stdout:Starting a Gradle Daemon (subsequent builds will be faster) 2019-12-11T16:46:35.031 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/spring/gradle/dependency-management-plugin/1.0.3.RELEASE/dependency-management-plugin-1.0.3.RELEASE.pom 2019-12-11T16:46:35.132 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/spring/gradle/dependency-management-plugin/1.0.3.RELEASE/dependency-management-plugin-1.0.3.RELEASE.jar 2019-12-11T16:46:36.232 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :clean UP-TO-DATE 2019-12-11T16:46:36.331 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-bom/1.11.549/aws-java-sdk-bom-1.11.549.pom 2019-12-11T16:46:36.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pom/1.11.549/aws-java-sdk-pom-1.11.549.pom 2019-12-11T16:46:36.634 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sqs/1.11.549/aws-java-sdk-sqs-1.11.549.pom 2019-12-11T16:46:36.634 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.549/aws-java-sdk-core-1.11.549.pom 2019-12-11T16:46:36.634 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk/1.11.549/aws-java-sdk-1.11.549.pom 2019-12-11T16:46:36.635 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.pom 2019-12-11T16:46:36.635 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-server/2.44.0/selenium-server-2.44.0.pom 2019-12-11T16:46:36.635 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.549/aws-java-sdk-s3-1.11.549.pom 2019-12-11T16:46:36.635 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/testng/testng/6.1.1/testng-6.1.1.pom 2019-12-11T16:46:36.635 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/3/oss-parent-3.pom 2019-12-11T16:46:36.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-parent/2.44.0/selenium-parent-2.44.0.pom 2019-12-11T16:46:36.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.pom 2019-12-11T16:46:36.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.pom 2019-12-11T16:46:36.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/joda-time/joda-time/2.8.1/joda-time-2.8.1.pom 2019-12-11T16:46:36.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.7.2/jackson-databind-2.6.7.2.pom 2019-12-11T16:46:36.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.pom 2019-12-11T16:46:36.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.pom 2019-12-11T16:46:36.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.6.2/jackson-parent-2.6.2.pom 2019-12-11T16:46:36.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/28/commons-parent-28.pom 2019-12-11T16:46:36.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.5.5/httpcomponents-client-4.5.5.pom 2019-12-11T16:46:36.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/oss-parent/24/oss-parent-24.pom 2019-12-11T16:46:36.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-parent/10/httpcomponents-parent-10.pom 2019-12-11T16:46:36.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom 2019-12-11T16:46:36.934 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom 2019-12-11T16:46:36.934 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-managedblockchain/1.11.549/aws-java-sdk-managedblockchain-1.11.549.pom 2019-12-11T16:46:36.934 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-backup/1.11.549/aws-java-sdk-backup-1.11.549.pom 2019-12-11T16:46:36.934 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewaymanagementapi/1.11.549/aws-java-sdk-apigatewaymanagementapi-1.11.549.pom 2019-12-11T16:46:36.934 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-worklink/1.11.549/aws-java-sdk-worklink-1.11.549.pom 2019-12-11T16:46:36.935 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-textract/1.11.549/aws-java-sdk-textract-1.11.549.pom 2019-12-11T16:46:36.935 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kafka/1.11.549/aws-java-sdk-kafka-1.11.549.pom 2019-12-11T16:46:36.935 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewayv2/1.11.549/aws-java-sdk-apigatewayv2-1.11.549.pom 2019-12-11T16:46:36.935 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-docdb/1.11.549/aws-java-sdk-docdb-1.11.549.pom 2019-12-11T16:46:36.935 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisanalyticsv2/1.11.549/aws-java-sdk-kinesisanalyticsv2-1.11.549.pom 2019-12-11T16:46:36.935 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehendmedical/1.11.549/aws-java-sdk-comprehendmedical-1.11.549.pom 2019-12-11T16:46:36.935 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fsx/1.11.549/aws-java-sdk-fsx-1.11.549.pom 2019-12-11T16:46:36.936 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-securityhub/1.11.549/aws-java-sdk-securityhub-1.11.549.pom 2019-12-11T16:46:36.936 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-licensemanager/1.11.549/aws-java-sdk-licensemanager-1.11.549.pom 2019-12-11T16:46:36.936 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-globalaccelerator/1.11.549/aws-java-sdk-globalaccelerator-1.11.549.pom 2019-12-11T16:46:36.936 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconnect/1.11.549/aws-java-sdk-mediaconnect-1.11.549.pom 2019-12-11T16:46:37.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-robomaker/1.11.549/aws-java-sdk-robomaker-1.11.549.pom 2019-12-11T16:46:37.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-quicksight/1.11.549/aws-java-sdk-quicksight-1.11.549.pom 2019-12-11T16:46:37.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-amplify/1.11.549/aws-java-sdk-amplify-1.11.549.pom 2019-12-11T16:46:37.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53resolver/1.11.549/aws-java-sdk-route53resolver-1.11.549.pom 2019-12-11T16:46:37.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transfer/1.11.549/aws-java-sdk-transfer-1.11.549.pom 2019-12-11T16:46:37.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ram/1.11.549/aws-java-sdk-ram-1.11.549.pom 2019-12-11T16:46:37.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datasync/1.11.549/aws-java-sdk-datasync-1.11.549.pom 2019-12-11T16:46:37.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3control/1.11.549/aws-java-sdk-s3control-1.11.549.pom 2019-12-11T16:46:37.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rdsdata/1.11.549/aws-java-sdk-rdsdata-1.11.549.pom 2019-12-11T16:46:37.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointsmsvoice/1.11.549/aws-java-sdk-pinpointsmsvoice-1.11.549.pom 2019-12-11T16:46:37.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointemail/1.11.549/aws-java-sdk-pinpointemail-1.11.549.pom 2019-12-11T16:46:37.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-chime/1.11.549/aws-java-sdk-chime-1.11.549.pom 2019-12-11T16:46:37.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dlm/1.11.549/aws-java-sdk-dlm-1.11.549.pom 2019-12-11T16:46:37.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-macie/1.11.549/aws-java-sdk-macie-1.11.549.pom 2019-12-11T16:46:37.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-signer/1.11.549/aws-java-sdk-signer-1.11.549.pom 2019-12-11T16:46:37.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-eks/1.11.549/aws-java-sdk-eks-1.11.549.pom 2019-12-11T16:46:37.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediatailor/1.11.549/aws-java-sdk-mediatailor-1.11.549.pom 2019-12-11T16:46:37.037 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-neptune/1.11.549/aws-java-sdk-neptune-1.11.549.pom 2019-12-11T16:46:37.037 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickprojects/1.11.549/aws-java-sdk-iot1clickprojects-1.11.549.pom 2019-12-11T16:46:37.038 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickdevices/1.11.549/aws-java-sdk-iot1clickdevices-1.11.549.pom 2019-12-11T16:46:37.039 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pi/1.11.549/aws-java-sdk-pi-1.11.549.pom 2019-12-11T16:46:37.039 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotanalytics/1.11.549/aws-java-sdk-iotanalytics-1.11.549.pom 2019-12-11T16:46:37.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fms/1.11.549/aws-java-sdk-fms-1.11.549.pom 2019-12-11T16:46:37.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-connect/1.11.549/aws-java-sdk-connect-1.11.549.pom 2019-12-11T16:46:37.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workmail/1.11.549/aws-java-sdk-workmail-1.11.549.pom 2019-12-11T16:46:37.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-secretsmanager/1.11.549/aws-java-sdk-secretsmanager-1.11.549.pom 2019-12-11T16:46:37.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acmpca/1.11.549/aws-java-sdk-acmpca-1.11.549.pom 2019-12-11T16:46:37.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscalingplans/1.11.549/aws-java-sdk-autoscalingplans-1.11.549.pom 2019-12-11T16:46:37.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicediscovery/1.11.549/aws-java-sdk-servicediscovery-1.11.549.pom 2019-12-11T16:46:37.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-alexaforbusiness/1.11.549/aws-java-sdk-alexaforbusiness-1.11.549.pom 2019-12-11T16:46:37.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-serverlessapplicationrepository/1.11.549/aws-java-sdk-serverlessapplicationrepository-1.11.549.pom 2019-12-11T16:46:37.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transcribe/1.11.549/aws-java-sdk-transcribe-1.11.549.pom 2019-12-11T16:46:37.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloud9/1.11.549/aws-java-sdk-cloud9-1.11.549.pom 2019-12-11T16:46:37.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroups/1.11.549/aws-java-sdk-resourcegroups-1.11.549.pom 2019-12-11T16:46:37.139 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehend/1.11.549/aws-java-sdk-comprehend-1.11.549.pom 2019-12-11T16:46:37.140 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemaker/1.11.549/aws-java-sdk-sagemaker-1.11.549.pom 2019-12-11T16:46:37.140 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-translate/1.11.549/aws-java-sdk-translate-1.11.549.pom 2019-12-11T16:46:37.140 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemakerruntime/1.11.549/aws-java-sdk-sagemakerruntime-1.11.549.pom 2019-12-11T16:46:37.140 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appsync/1.11.549/aws-java-sdk-appsync-1.11.549.pom 2019-12-11T16:46:37.140 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-guardduty/1.11.549/aws-java-sdk-guardduty-1.11.549.pom 2019-12-11T16:46:37.140 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisvideo/1.11.549/aws-java-sdk-kinesisvideo-1.11.549.pom 2019-12-11T16:46:37.141 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mq/1.11.549/aws-java-sdk-mq-1.11.549.pom 2019-12-11T16:46:37.141 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconvert/1.11.549/aws-java-sdk-mediaconvert-1.11.549.pom 2019-12-11T16:46:37.142 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastore/1.11.549/aws-java-sdk-mediastore-1.11.549.pom 2019-12-11T16:46:37.142 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediapackage/1.11.549/aws-java-sdk-mediapackage-1.11.549.pom 2019-12-11T16:46:37.142 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-medialive/1.11.549/aws-java-sdk-medialive-1.11.549.pom 2019-12-11T16:46:37.142 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotjobsdataplane/1.11.549/aws-java-sdk-iotjobsdataplane-1.11.549.pom 2019-12-11T16:46:37.142 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastoredata/1.11.549/aws-java-sdk-mediastoredata-1.11.549.pom 2019-12-11T16:46:37.143 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costexplorer/1.11.549/aws-java-sdk-costexplorer-1.11.549.pom 2019-12-11T16:46:37.143 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mobile/1.11.549/aws-java-sdk-mobile-1.11.549.pom 2019-12-11T16:46:37.143 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pricing/1.11.549/aws-java-sdk-pricing-1.11.549.pom 2019-12-11T16:46:37.143 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glue/1.11.549/aws-java-sdk-glue-1.11.549.pom 2019-12-11T16:46:37.143 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsmv2/1.11.549/aws-java-sdk-cloudhsmv2-1.11.549.pom 2019-12-11T16:46:37.143 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-migrationhub/1.11.549/aws-java-sdk-migrationhub-1.11.549.pom 2019-12-11T16:46:37.143 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dax/1.11.549/aws-java-sdk-dax-1.11.549.pom 2019-12-11T16:46:37.144 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-greengrass/1.11.549/aws-java-sdk-greengrass-1.11.549.pom 2019-12-11T16:46:37.144 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplaceentitlement/1.11.549/aws-java-sdk-marketplaceentitlement-1.11.549.pom 2019-12-11T16:46:37.144 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codestar/1.11.549/aws-java-sdk-codestar-1.11.549.pom 2019-12-11T16:46:37.144 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroupstaggingapi/1.11.549/aws-java-sdk-resourcegroupstaggingapi-1.11.549.pom 2019-12-11T16:46:37.144 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lexmodelbuilding/1.11.549/aws-java-sdk-lexmodelbuilding-1.11.549.pom 2019-12-11T16:46:37.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpoint/1.11.549/aws-java-sdk-pinpoint-1.11.549.pom 2019-12-11T16:46:37.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-xray/1.11.549/aws-java-sdk-xray-1.11.549.pom 2019-12-11T16:46:37.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-athena/1.11.549/aws-java-sdk-athena-1.11.549.pom 2019-12-11T16:46:37.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworkscm/1.11.549/aws-java-sdk-opsworkscm-1.11.549.pom 2019-12-11T16:46:37.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-support/1.11.549/aws-java-sdk-support-1.11.549.pom 2019-12-11T16:46:37.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-storagegateway/1.11.549/aws-java-sdk-storagegateway-1.11.549.pom 2019-12-11T16:46:37.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servermigration/1.11.549/aws-java-sdk-servermigration-1.11.549.pom 2019-12-11T16:46:37.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicecatalog/1.11.549/aws-java-sdk-servicecatalog-1.11.549.pom 2019-12-11T16:46:37.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpledb/1.11.549/aws-java-sdk-simpledb-1.11.549.pom 2019-12-11T16:46:37.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpleworkflow/1.11.549/aws-java-sdk-simpleworkflow-1.11.549.pom 2019-12-11T16:46:37.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53/1.11.549/aws-java-sdk-route53-1.11.549.pom 2019-12-11T16:46:37.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sts/1.11.549/aws-java-sdk-sts-1.11.549.pom 2019-12-11T16:46:37.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-importexport/1.11.549/aws-java-sdk-importexport-1.11.549.pom 2019-12-11T16:46:37.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rds/1.11.549/aws-java-sdk-rds-1.11.549.pom 2019-12-11T16:46:37.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-redshift/1.11.549/aws-java-sdk-redshift-1.11.549.pom 2019-12-11T16:46:37.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glacier/1.11.549/aws-java-sdk-glacier-1.11.549.pom 2019-12-11T16:46:37.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticbeanstalk/1.11.549/aws-java-sdk-elasticbeanstalk-1.11.549.pom 2019-12-11T16:46:37.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iam/1.11.549/aws-java-sdk-iam-1.11.549.pom 2019-12-11T16:46:37.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datapipeline/1.11.549/aws-java-sdk-datapipeline-1.11.549.pom 2019-12-11T16:46:37.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancingv2/1.11.549/aws-java-sdk-elasticloadbalancingv2-1.11.549.pom 2019-12-11T16:46:37.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-emr/1.11.549/aws-java-sdk-emr-1.11.549.pom 2019-12-11T16:46:37.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancing/1.11.549/aws-java-sdk-elasticloadbalancing-1.11.549.pom 2019-12-11T16:46:37.241 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticache/1.11.549/aws-java-sdk-elasticache-1.11.549.pom 2019-12-11T16:46:37.241 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ec2/1.11.549/aws-java-sdk-ec2-1.11.549.pom 2019-12-11T16:46:37.242 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elastictranscoder/1.11.549/aws-java-sdk-elastictranscoder-1.11.549.pom 2019-12-11T16:46:37.242 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dynamodb/1.11.549/aws-java-sdk-dynamodb-1.11.549.pom 2019-12-11T16:46:37.242 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-budgets/1.11.549/aws-java-sdk-budgets-1.11.549.pom 2019-12-11T16:46:37.242 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-events/1.11.549/aws-java-sdk-events-1.11.549.pom 2019-12-11T16:46:37.242 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidentity/1.11.549/aws-java-sdk-cognitoidentity-1.11.549.pom 2019-12-11T16:46:37.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatch/1.11.549/aws-java-sdk-cloudwatch-1.11.549.pom 2019-12-11T16:46:37.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-logs/1.11.549/aws-java-sdk-logs-1.11.549.pom 2019-12-11T16:46:37.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudtrail/1.11.549/aws-java-sdk-cloudtrail-1.11.549.pom 2019-12-11T16:46:37.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sns/1.11.549/aws-java-sdk-sns-1.11.549.pom 2019-12-11T16:46:37.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directconnect/1.11.549/aws-java-sdk-directconnect-1.11.549.pom 2019-12-11T16:46:37.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitosync/1.11.549/aws-java-sdk-cognitosync-1.11.549.pom 2019-12-11T16:46:37.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudformation/1.11.549/aws-java-sdk-cloudformation-1.11.549.pom 2019-12-11T16:46:37.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-clouddirectory/1.11.549/aws-java-sdk-clouddirectory-1.11.549.pom 2019-12-11T16:46:37.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesis/1.11.549/aws-java-sdk-kinesis-1.11.549.pom 2019-12-11T16:46:37.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudfront/1.11.549/aws-java-sdk-cloudfront-1.11.549.pom 2019-12-11T16:46:37.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworks/1.11.549/aws-java-sdk-opsworks-1.11.549.pom 2019-12-11T16:46:37.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ses/1.11.549/aws-java-sdk-ses-1.11.549.pom 2019-12-11T16:46:37.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscaling/1.11.549/aws-java-sdk-autoscaling-1.11.549.pom 2019-12-11T16:46:37.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudsearch/1.11.549/aws-java-sdk-cloudsearch-1.11.549.pom 2019-12-11T16:46:37.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatchmetrics/1.11.549/aws-java-sdk-cloudwatchmetrics-1.11.549.pom 2019-12-11T16:46:37.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codedeploy/1.11.549/aws-java-sdk-codedeploy-1.11.549.pom 2019-12-11T16:46:37.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.549/aws-java-sdk-kms-1.11.549.pom 2019-12-11T16:46:37.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codepipeline/1.11.549/aws-java-sdk-codepipeline-1.11.549.pom 2019-12-11T16:46:37.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-config/1.11.549/aws-java-sdk-config-1.11.549.pom 2019-12-11T16:46:37.336 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecr/1.11.549/aws-java-sdk-ecr-1.11.549.pom 2019-12-11T16:46:37.336 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecs/1.11.549/aws-java-sdk-ecs-1.11.549.pom 2019-12-11T16:46:37.336 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ssm/1.11.549/aws-java-sdk-ssm-1.11.549.pom 2019-12-11T16:46:37.336 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-machinelearning/1.11.549/aws-java-sdk-machinelearning-1.11.549.pom 2019-12-11T16:46:37.336 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workspaces/1.11.549/aws-java-sdk-workspaces-1.11.549.pom 2019-12-11T16:46:37.336 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsm/1.11.549/aws-java-sdk-cloudhsm-1.11.549.pom 2019-12-11T16:46:37.336 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lambda/1.11.549/aws-java-sdk-lambda-1.11.549.pom 2019-12-11T16:46:37.337 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directory/1.11.549/aws-java-sdk-directory-1.11.549.pom 2019-12-11T16:46:37.337 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-efs/1.11.549/aws-java-sdk-efs-1.11.549.pom 2019-12-11T16:46:37.337 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codecommit/1.11.549/aws-java-sdk-codecommit-1.11.549.pom 2019-12-11T16:46:37.337 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticsearch/1.11.549/aws-java-sdk-elasticsearch-1.11.549.pom 2019-12-11T16:46:37.337 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacecommerceanalytics/1.11.549/aws-java-sdk-marketplacecommerceanalytics-1.11.549.pom 2019-12-11T16:46:37.337 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot/1.11.549/aws-java-sdk-iot-1.11.549.pom 2019-12-11T16:46:37.337 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-inspector/1.11.549/aws-java-sdk-inspector-1.11.549.pom 2019-12-11T16:46:37.338 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-waf/1.11.549/aws-java-sdk-waf-1.11.549.pom 2019-12-11T16:46:37.338 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-devicefarm/1.11.549/aws-java-sdk-devicefarm-1.11.549.pom 2019-12-11T16:46:37.338 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-gamelift/1.11.549/aws-java-sdk-gamelift-1.11.549.pom 2019-12-11T16:46:37.338 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-api-gateway/1.11.549/aws-java-sdk-api-gateway-1.11.549.pom 2019-12-11T16:46:37.338 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acm/1.11.549/aws-java-sdk-acm-1.11.549.pom 2019-12-11T16:46:37.338 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dms/1.11.549/aws-java-sdk-dms-1.11.549.pom 2019-12-11T16:46:37.338 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacemeteringservice/1.11.549/aws-java-sdk-marketplacemeteringservice-1.11.549.pom 2019-12-11T16:46:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidp/1.11.549/aws-java-sdk-cognitoidp-1.11.549.pom 2019-12-11T16:46:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-discovery/1.11.549/aws-java-sdk-discovery-1.11.549.pom 2019-12-11T16:46:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-applicationautoscaling/1.11.549/aws-java-sdk-applicationautoscaling-1.11.549.pom 2019-12-11T16:46:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-snowball/1.11.549/aws-java-sdk-snowball-1.11.549.pom 2019-12-11T16:46:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rekognition/1.11.549/aws-java-sdk-rekognition-1.11.549.pom 2019-12-11T16:46:37.339 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-polly/1.11.549/aws-java-sdk-polly-1.11.549.pom 2019-12-11T16:46:37.431 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-stepfunctions/1.11.549/aws-java-sdk-stepfunctions-1.11.549.pom 2019-12-11T16:46:37.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-health/1.11.549/aws-java-sdk-health-1.11.549.pom 2019-12-11T16:46:37.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costandusagereport/1.11.549/aws-java-sdk-costandusagereport-1.11.549.pom 2019-12-11T16:46:37.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appstream/1.11.549/aws-java-sdk-appstream-1.11.549.pom 2019-12-11T16:46:37.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codebuild/1.11.549/aws-java-sdk-codebuild-1.11.549.pom 2019-12-11T16:46:37.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-shield/1.11.549/aws-java-sdk-shield-1.11.549.pom 2019-12-11T16:46:37.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lightsail/1.11.549/aws-java-sdk-lightsail-1.11.549.pom 2019-12-11T16:46:37.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-batch/1.11.549/aws-java-sdk-batch-1.11.549.pom 2019-12-11T16:46:37.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lex/1.11.549/aws-java-sdk-lex-1.11.549.pom 2019-12-11T16:46:37.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mechanicalturkrequester/1.11.549/aws-java-sdk-mechanicalturkrequester-1.11.549.pom 2019-12-11T16:46:37.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-organizations/1.11.549/aws-java-sdk-organizations-1.11.549.pom 2019-12-11T16:46:37.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workdocs/1.11.549/aws-java-sdk-workdocs-1.11.549.pom 2019-12-11T16:46:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-models/1.11.549/aws-java-sdk-models-1.11.549.pom 2019-12-11T16:46:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-swf-libraries/1.11.22/aws-java-sdk-swf-libraries-1.11.22.pom 2019-12-11T16:46:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appmesh/1.11.549/aws-java-sdk-appmesh-1.11.549.pom 2019-12-11T16:46:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pom/1.11.22/aws-java-sdk-pom-1.11.22.pom 2019-12-11T16:46:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/jmespath-java/1.11.549/jmespath-java-1.11.549.pom 2019-12-11T16:46:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.pom 2019-12-11T16:46:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.48/bcpkix-jdk15on-1.48.pom 2019-12-11T16:46:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-java/2.44.0/selenium-java-2.44.0.pom 2019-12-11T16:46:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-repacked/7.6.1/jetty-repacked-7.6.1.pom 2019-12-11T16:46:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-rc-repacked/5/jetty-rc-repacked-5.pom 2019-12-11T16:46:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.8/snakeyaml-1.8.pom 2019-12-11T16:46:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.48/bcprov-jdk15on-1.48.pom 2019-12-11T16:46:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.pom 2019-12-11T16:46:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/mortbay/jetty/servlet-api-2.5/6.1.9/servlet-api-2.5-6.1.9.pom 2019-12-11T16:46:37.531 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/32/commons-parent-32.pom 2019-12-11T16:46:37.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/mortbay/jetty/project/6.1.9/project-6.1.9.pom 2019-12-11T16:46:37.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/mx4j/mx4j-tools/3.0.1/mx4j-tools-3.0.1.pom 2019-12-11T16:46:37.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/junit/junit/3.8.1/junit-3.8.1.pom 2019-12-11T16:46:37.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/beanshell/bsh/2.0b4/bsh-2.0b4.pom 2019-12-11T16:46:37.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/beust/jcommander/1.12/jcommander-1.12.pom 2019-12-11T16:46:37.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/beanshell/beanshell/2.0b4/beanshell-2.0b4.pom 2019-12-11T16:46:37.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.pom 2019-12-11T16:46:37.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/34/commons-parent-34.pom 2019-12-11T16:46:37.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.pom 2019-12-11T16:46:37.634 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35.pom 2019-12-11T16:46:37.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/15/apache-15.pom 2019-12-11T16:46:37.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.pom 2019-12-11T16:46:37.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-core/4.4.9/httpcomponents-core-4.4.9.pom 2019-12-11T16:46:37.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-parent/9/httpcomponents-parent-9.pom 2019-12-11T16:46:37.733 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.pom 2019-12-11T16:46:37.733 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.pom 2019-12-11T16:46:37.733 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.6.1/jackson-parent-2.6.1.pom 2019-12-11T16:46:37.733 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/oss-parent/23/oss-parent-23.pom 2019-12-11T16:46:37.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.17.Final/netty-handler-4.1.17.Final.pom 2019-12-11T16:46:37.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec-http/4.1.17.Final/netty-codec-http-4.1.17.Final.pom 2019-12-11T16:46:37.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-parent/4.1.17.Final/netty-parent-4.1.17.Final.pom 2019-12-11T16:46:37.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom 2019-12-11T16:46:37.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-firefox-driver/2.44.0/selenium-firefox-driver-2.44.0.pom 2019-12-11T16:46:37.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-safari-driver/2.44.0/selenium-safari-driver-2.44.0.pom 2019-12-11T16:46:37.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-chrome-driver/2.44.0/selenium-chrome-driver-2.44.0.pom 2019-12-11T16:46:37.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/webbitserver/webbit/0.4.14/webbit-0.4.14.pom 2019-12-11T16:46:37.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-ie-driver/2.44.0/selenium-ie-driver-2.44.0.pom 2019-12-11T16:46:37.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-support/2.44.0/selenium-support-2.44.0.pom 2019-12-11T16:46:37.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-htmlunit-driver/2.44.0/selenium-htmlunit-driver-2.44.0.pom 2019-12-11T16:46:37.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/6/oss-parent-6.pom 2019-12-11T16:46:37.834 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec/4.1.17.Final/netty-codec-4.1.17.Final.pom 2019-12-11T16:46:37.931 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-transport/4.1.17.Final/netty-transport-4.1.17.Final.pom 2019-12-11T16:46:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.17.Final/netty-buffer-4.1.17.Final.pom 2019-12-11T16:46:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/2.44.0/selenium-remote-driver-2.44.0.pom 2019-12-11T16:46:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit/2.15/htmlunit-2.15.pom 2019-12-11T16:46:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.pom 2019-12-11T16:46:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.pom 2019-12-11T16:46:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/25/commons-parent-25.pom 2019-12-11T16:46:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/17/commons-parent-17.pom 2019-12-11T16:46:37.933 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/7/apache-7.pom 2019-12-11T16:46:37.933 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/9/apache-9.pom 2019-12-11T16:46:38.032 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/platform/3.4.0/platform-3.4.0.pom 2019-12-11T16:46:38.032 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/jna/3.4.0/jna-3.4.0.pom 2019-12-11T16:46:38.032 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty/3.5.2.Final/netty-3.5.2.Final.pom 2019-12-11T16:46:38.032 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/7/oss-parent-7.pom 2019-12-11T16:46:38.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-common/4.1.17.Final/netty-common-4.1.17.Final.pom 2019-12-11T16:46:38.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.17.Final/netty-resolver-4.1.17.Final.pom 2019-12-11T16:46:38.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/google/guava/guava/18.0/guava-18.0.pom 2019-12-11T16:46:38.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.3/gson-2.3.pom 2019-12-11T16:46:38.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/cglib/cglib-nodep/2.1_3/cglib-nodep-2.1_3.pom 2019-12-11T16:46:38.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-api/2.44.0/selenium-api-2.44.0.pom 2019-12-11T16:46:38.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/18.0/guava-parent-18.0.pom 2019-12-11T16:46:38.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit-core-js/2.15/htmlunit-core-js-2.15.pom 2019-12-11T16:46:38.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.3.3/httpmime-4.3.3.pom 2019-12-11T16:46:38.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xalan/xalan/2.7.1/xalan-2.7.1.pom 2019-12-11T16:46:38.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.pom 2019-12-11T16:46:38.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/cssparser/cssparser/0.9.14/cssparser-0.9.14.pom 2019-12-11T16:46:38.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.pom 2019-12-11T16:46:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.11.0/xercesImpl-2.11.0.pom 2019-12-11T16:46:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/nekohtml/nekohtml/1.9.21/nekohtml-1.9.21.pom 2019-12-11T16:46:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/9/commons-parent-9.pom 2019-12-11T16:46:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/33/commons-parent-33.pom 2019-12-11T16:46:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-websocket/8.1.15.v20140411/jetty-websocket-8.1.15.v20140411.pom 2019-12-11T16:46:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.3.3/httpcomponents-client-4.3.3.pom 2019-12-11T16:46:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/4/apache-4.pom 2019-12-11T16:46:38.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-project/8.1.15.v20140411/jetty-project-8.1.15.v20140411.pom 2019-12-11T16:46:38.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/project/7/project-7.pom 2019-12-11T16:46:38.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-parent/20/jetty-parent-20.pom 2019-12-11T16:46:38.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xalan/serializer/2.7.1/serializer-2.7.1.pom 2019-12-11T16:46:38.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.pom 2019-12-11T16:46:38.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/w3c/css/sac/1.3/sac-1.3.pom 2019-12-11T16:46:38.231 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/8.1.15.v20140411/jetty-io-8.1.15.v20140411.pom 2019-12-11T16:46:38.231 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/8.1.15.v20140411/jetty-util-8.1.15.v20140411.pom 2019-12-11T16:46:38.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/8.1.15.v20140411/jetty-http-8.1.15.v20140411.pom 2019-12-11T16:46:39.032 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3control/1.11.549/aws-java-sdk-s3control-1.11.549.jar 2019-12-11T16:46:39.032 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatchmetrics/1.11.549/aws-java-sdk-cloudwatchmetrics-1.11.549.jar 2019-12-11T16:46:39.032 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sqs/1.11.549/aws-java-sdk-sqs-1.11.549.jar 2019-12-11T16:46:39.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sns/1.11.549/aws-java-sdk-sns-1.11.549.jar 2019-12-11T16:46:39.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk/1.11.549/aws-java-sdk-1.11.549.jar 2019-12-11T16:46:39.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glacier/1.11.549/aws-java-sdk-glacier-1.11.549.jar 2019-12-11T16:46:39.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.549/aws-java-sdk-s3-1.11.549.jar 2019-12-11T16:46:39.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-managedblockchain/1.11.549/aws-java-sdk-managedblockchain-1.11.549.jar 2019-12-11T16:46:39.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dynamodb/1.11.549/aws-java-sdk-dynamodb-1.11.549.jar 2019-12-11T16:46:39.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-textract/1.11.549/aws-java-sdk-textract-1.11.549.jar 2019-12-11T16:46:39.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-worklink/1.11.549/aws-java-sdk-worklink-1.11.549.jar 2019-12-11T16:46:39.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-backup/1.11.549/aws-java-sdk-backup-1.11.549.jar 2019-12-11T16:46:39.132 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kafka/1.11.549/aws-java-sdk-kafka-1.11.549.jar 2019-12-11T16:46:39.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-docdb/1.11.549/aws-java-sdk-docdb-1.11.549.jar 2019-12-11T16:46:39.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewaymanagementapi/1.11.549/aws-java-sdk-apigatewaymanagementapi-1.11.549.jar 2019-12-11T16:46:39.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewayv2/1.11.549/aws-java-sdk-apigatewayv2-1.11.549.jar 2019-12-11T16:46:39.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-licensemanager/1.11.549/aws-java-sdk-licensemanager-1.11.549.jar 2019-12-11T16:46:39.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-securityhub/1.11.549/aws-java-sdk-securityhub-1.11.549.jar 2019-12-11T16:46:39.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appmesh/1.11.549/aws-java-sdk-appmesh-1.11.549.jar 2019-12-11T16:46:39.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fsx/1.11.549/aws-java-sdk-fsx-1.11.549.jar 2019-12-11T16:46:39.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconnect/1.11.549/aws-java-sdk-mediaconnect-1.11.549.jar 2019-12-11T16:46:39.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehendmedical/1.11.549/aws-java-sdk-comprehendmedical-1.11.549.jar 2019-12-11T16:46:39.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisanalyticsv2/1.11.549/aws-java-sdk-kinesisanalyticsv2-1.11.549.jar 2019-12-11T16:46:39.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-globalaccelerator/1.11.549/aws-java-sdk-globalaccelerator-1.11.549.jar 2019-12-11T16:46:39.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transfer/1.11.549/aws-java-sdk-transfer-1.11.549.jar 2019-12-11T16:46:39.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datasync/1.11.549/aws-java-sdk-datasync-1.11.549.jar 2019-12-11T16:46:39.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-amplify/1.11.549/aws-java-sdk-amplify-1.11.549.jar 2019-12-11T16:46:39.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-robomaker/1.11.549/aws-java-sdk-robomaker-1.11.549.jar 2019-12-11T16:46:39.135 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rdsdata/1.11.549/aws-java-sdk-rdsdata-1.11.549.jar 2019-12-11T16:46:39.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-quicksight/1.11.549/aws-java-sdk-quicksight-1.11.549.jar 2019-12-11T16:46:39.136 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53resolver/1.11.549/aws-java-sdk-route53resolver-1.11.549.jar 2019-12-11T16:46:39.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointemail/1.11.549/aws-java-sdk-pinpointemail-1.11.549.jar 2019-12-11T16:46:39.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ram/1.11.549/aws-java-sdk-ram-1.11.549.jar 2019-12-11T16:46:39.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-signer/1.11.549/aws-java-sdk-signer-1.11.549.jar 2019-12-11T16:46:39.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointsmsvoice/1.11.549/aws-java-sdk-pinpointsmsvoice-1.11.549.jar 2019-12-11T16:46:39.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-chime/1.11.549/aws-java-sdk-chime-1.11.549.jar 2019-12-11T16:46:39.232 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-macie/1.11.549/aws-java-sdk-macie-1.11.549.jar 2019-12-11T16:46:39.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dlm/1.11.549/aws-java-sdk-dlm-1.11.549.jar 2019-12-11T16:46:39.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-eks/1.11.549/aws-java-sdk-eks-1.11.549.jar 2019-12-11T16:46:39.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediatailor/1.11.549/aws-java-sdk-mediatailor-1.11.549.jar 2019-12-11T16:46:39.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-neptune/1.11.549/aws-java-sdk-neptune-1.11.549.jar 2019-12-11T16:46:39.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pi/1.11.549/aws-java-sdk-pi-1.11.549.jar 2019-12-11T16:46:39.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickprojects/1.11.549/aws-java-sdk-iot1clickprojects-1.11.549.jar 2019-12-11T16:46:39.233 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickdevices/1.11.549/aws-java-sdk-iot1clickdevices-1.11.549.jar 2019-12-11T16:46:39.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acmpca/1.11.549/aws-java-sdk-acmpca-1.11.549.jar 2019-12-11T16:46:39.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotanalytics/1.11.549/aws-java-sdk-iotanalytics-1.11.549.jar 2019-12-11T16:46:39.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fms/1.11.549/aws-java-sdk-fms-1.11.549.jar 2019-12-11T16:46:39.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-connect/1.11.549/aws-java-sdk-connect-1.11.549.jar 2019-12-11T16:46:39.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transcribe/1.11.549/aws-java-sdk-transcribe-1.11.549.jar 2019-12-11T16:46:39.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscalingplans/1.11.549/aws-java-sdk-autoscalingplans-1.11.549.jar 2019-12-11T16:46:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-secretsmanager/1.11.549/aws-java-sdk-secretsmanager-1.11.549.jar 2019-12-11T16:46:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workmail/1.11.549/aws-java-sdk-workmail-1.11.549.jar 2019-12-11T16:46:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicediscovery/1.11.549/aws-java-sdk-servicediscovery-1.11.549.jar 2019-12-11T16:46:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloud9/1.11.549/aws-java-sdk-cloud9-1.11.549.jar 2019-12-11T16:46:39.331 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-serverlessapplicationrepository/1.11.549/aws-java-sdk-serverlessapplicationrepository-1.11.549.jar 2019-12-11T16:46:39.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-alexaforbusiness/1.11.549/aws-java-sdk-alexaforbusiness-1.11.549.jar 2019-12-11T16:46:39.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroups/1.11.549/aws-java-sdk-resourcegroups-1.11.549.jar 2019-12-11T16:46:39.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-translate/1.11.549/aws-java-sdk-translate-1.11.549.jar 2019-12-11T16:46:39.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehend/1.11.549/aws-java-sdk-comprehend-1.11.549.jar 2019-12-11T16:46:39.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotjobsdataplane/1.11.549/aws-java-sdk-iotjobsdataplane-1.11.549.jar 2019-12-11T16:46:39.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemakerruntime/1.11.549/aws-java-sdk-sagemakerruntime-1.11.549.jar 2019-12-11T16:46:39.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemaker/1.11.549/aws-java-sdk-sagemaker-1.11.549.jar 2019-12-11T16:46:39.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appsync/1.11.549/aws-java-sdk-appsync-1.11.549.jar 2019-12-11T16:46:39.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mq/1.11.549/aws-java-sdk-mq-1.11.549.jar 2019-12-11T16:46:39.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisvideo/1.11.549/aws-java-sdk-kinesisvideo-1.11.549.jar 2019-12-11T16:46:39.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconvert/1.11.549/aws-java-sdk-mediaconvert-1.11.549.jar 2019-12-11T16:46:39.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-guardduty/1.11.549/aws-java-sdk-guardduty-1.11.549.jar 2019-12-11T16:46:39.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastore/1.11.549/aws-java-sdk-mediastore-1.11.549.jar 2019-12-11T16:46:39.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastoredata/1.11.549/aws-java-sdk-mediastoredata-1.11.549.jar 2019-12-11T16:46:39.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediapackage/1.11.549/aws-java-sdk-mediapackage-1.11.549.jar 2019-12-11T16:46:39.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-medialive/1.11.549/aws-java-sdk-medialive-1.11.549.jar 2019-12-11T16:46:39.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pricing/1.11.549/aws-java-sdk-pricing-1.11.549.jar 2019-12-11T16:46:39.436 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mobile/1.11.549/aws-java-sdk-mobile-1.11.549.jar 2019-12-11T16:46:39.436 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsmv2/1.11.549/aws-java-sdk-cloudhsmv2-1.11.549.jar 2019-12-11T16:46:39.436 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costexplorer/1.11.549/aws-java-sdk-costexplorer-1.11.549.jar 2019-12-11T16:46:39.436 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dax/1.11.549/aws-java-sdk-dax-1.11.549.jar 2019-12-11T16:46:39.436 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glue/1.11.549/aws-java-sdk-glue-1.11.549.jar 2019-12-11T16:46:39.436 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-greengrass/1.11.549/aws-java-sdk-greengrass-1.11.549.jar 2019-12-11T16:46:39.437 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-migrationhub/1.11.549/aws-java-sdk-migrationhub-1.11.549.jar 2019-12-11T16:46:39.437 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplaceentitlement/1.11.549/aws-java-sdk-marketplaceentitlement-1.11.549.jar 2019-12-11T16:46:39.437 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-athena/1.11.549/aws-java-sdk-athena-1.11.549.jar 2019-12-11T16:46:39.437 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codestar/1.11.549/aws-java-sdk-codestar-1.11.549.jar 2019-12-11T16:46:39.437 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lexmodelbuilding/1.11.549/aws-java-sdk-lexmodelbuilding-1.11.549.jar 2019-12-11T16:46:39.437 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroupstaggingapi/1.11.549/aws-java-sdk-resourcegroupstaggingapi-1.11.549.jar 2019-12-11T16:46:39.437 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-xray/1.11.549/aws-java-sdk-xray-1.11.549.jar 2019-12-11T16:46:39.438 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpoint/1.11.549/aws-java-sdk-pinpoint-1.11.549.jar 2019-12-11T16:46:39.438 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworkscm/1.11.549/aws-java-sdk-opsworkscm-1.11.549.jar 2019-12-11T16:46:39.438 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-support/1.11.549/aws-java-sdk-support-1.11.549.jar 2019-12-11T16:46:39.531 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpledb/1.11.549/aws-java-sdk-simpledb-1.11.549.jar 2019-12-11T16:46:39.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servermigration/1.11.549/aws-java-sdk-servermigration-1.11.549.jar 2019-12-11T16:46:39.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicecatalog/1.11.549/aws-java-sdk-servicecatalog-1.11.549.jar 2019-12-11T16:46:39.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-swf-libraries/1.11.22/aws-java-sdk-swf-libraries-1.11.22.jar 2019-12-11T16:46:39.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53/1.11.549/aws-java-sdk-route53-1.11.549.jar 2019-12-11T16:46:39.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpleworkflow/1.11.549/aws-java-sdk-simpleworkflow-1.11.549.jar 2019-12-11T16:46:39.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-storagegateway/1.11.549/aws-java-sdk-storagegateway-1.11.549.jar 2019-12-11T16:46:39.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sts/1.11.549/aws-java-sdk-sts-1.11.549.jar 2019-12-11T16:46:39.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-importexport/1.11.549/aws-java-sdk-importexport-1.11.549.jar 2019-12-11T16:46:39.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rds/1.11.549/aws-java-sdk-rds-1.11.549.jar 2019-12-11T16:46:39.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-redshift/1.11.549/aws-java-sdk-redshift-1.11.549.jar 2019-12-11T16:46:39.631 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datapipeline/1.11.549/aws-java-sdk-datapipeline-1.11.549.jar 2019-12-11T16:46:39.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancing/1.11.549/aws-java-sdk-elasticloadbalancing-1.11.549.jar 2019-12-11T16:46:39.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancingv2/1.11.549/aws-java-sdk-elasticloadbalancingv2-1.11.549.jar 2019-12-11T16:46:39.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticbeanstalk/1.11.549/aws-java-sdk-elasticbeanstalk-1.11.549.jar 2019-12-11T16:46:39.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iam/1.11.549/aws-java-sdk-iam-1.11.549.jar 2019-12-11T16:46:39.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-emr/1.11.549/aws-java-sdk-emr-1.11.549.jar 2019-12-11T16:46:39.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elastictranscoder/1.11.549/aws-java-sdk-elastictranscoder-1.11.549.jar 2019-12-11T16:46:39.632 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticache/1.11.549/aws-java-sdk-elasticache-1.11.549.jar 2019-12-11T16:46:39.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-budgets/1.11.549/aws-java-sdk-budgets-1.11.549.jar 2019-12-11T16:46:39.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudtrail/1.11.549/aws-java-sdk-cloudtrail-1.11.549.jar 2019-12-11T16:46:39.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ec2/1.11.549/aws-java-sdk-ec2-1.11.549.jar 2019-12-11T16:46:39.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-logs/1.11.549/aws-java-sdk-logs-1.11.549.jar 2019-12-11T16:46:39.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatch/1.11.549/aws-java-sdk-cloudwatch-1.11.549.jar 2019-12-11T16:46:39.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidentity/1.11.549/aws-java-sdk-cognitoidentity-1.11.549.jar 2019-12-11T16:46:39.633 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-events/1.11.549/aws-java-sdk-events-1.11.549.jar 2019-12-11T16:46:39.634 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitosync/1.11.549/aws-java-sdk-cognitosync-1.11.549.jar 2019-12-11T16:46:39.731 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directconnect/1.11.549/aws-java-sdk-directconnect-1.11.549.jar 2019-12-11T16:46:39.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudfront/1.11.549/aws-java-sdk-cloudfront-1.11.549.jar 2019-12-11T16:46:39.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudformation/1.11.549/aws-java-sdk-cloudformation-1.11.549.jar 2019-12-11T16:46:39.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-clouddirectory/1.11.549/aws-java-sdk-clouddirectory-1.11.549.jar 2019-12-11T16:46:39.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesis/1.11.549/aws-java-sdk-kinesis-1.11.549.jar 2019-12-11T16:46:39.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ses/1.11.549/aws-java-sdk-ses-1.11.549.jar 2019-12-11T16:46:39.732 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworks/1.11.549/aws-java-sdk-opsworks-1.11.549.jar 2019-12-11T16:46:39.733 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscaling/1.11.549/aws-java-sdk-autoscaling-1.11.549.jar 2019-12-11T16:46:39.733 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudsearch/1.11.549/aws-java-sdk-cloudsearch-1.11.549.jar 2019-12-11T16:46:39.733 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codedeploy/1.11.549/aws-java-sdk-codedeploy-1.11.549.jar 2019-12-11T16:46:39.831 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codepipeline/1.11.549/aws-java-sdk-codepipeline-1.11.549.jar 2019-12-11T16:46:39.831 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.549/aws-java-sdk-kms-1.11.549.jar 2019-12-11T16:46:39.831 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-config/1.11.549/aws-java-sdk-config-1.11.549.jar 2019-12-11T16:46:39.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lambda/1.11.549/aws-java-sdk-lambda-1.11.549.jar 2019-12-11T16:46:39.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecs/1.11.549/aws-java-sdk-ecs-1.11.549.jar 2019-12-11T16:46:39.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsm/1.11.549/aws-java-sdk-cloudhsm-1.11.549.jar 2019-12-11T16:46:39.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecr/1.11.549/aws-java-sdk-ecr-1.11.549.jar 2019-12-11T16:46:39.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ssm/1.11.549/aws-java-sdk-ssm-1.11.549.jar 2019-12-11T16:46:39.832 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workspaces/1.11.549/aws-java-sdk-workspaces-1.11.549.jar 2019-12-11T16:46:39.833 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-machinelearning/1.11.549/aws-java-sdk-machinelearning-1.11.549.jar 2019-12-11T16:46:39.931 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directory/1.11.549/aws-java-sdk-directory-1.11.549.jar 2019-12-11T16:46:39.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-efs/1.11.549/aws-java-sdk-efs-1.11.549.jar 2019-12-11T16:46:39.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codecommit/1.11.549/aws-java-sdk-codecommit-1.11.549.jar 2019-12-11T16:46:39.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-devicefarm/1.11.549/aws-java-sdk-devicefarm-1.11.549.jar 2019-12-11T16:46:39.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticsearch/1.11.549/aws-java-sdk-elasticsearch-1.11.549.jar 2019-12-11T16:46:39.932 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacecommerceanalytics/1.11.549/aws-java-sdk-marketplacecommerceanalytics-1.11.549.jar 2019-12-11T16:46:39.933 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-waf/1.11.549/aws-java-sdk-waf-1.11.549.jar 2019-12-11T16:46:39.933 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-inspector/1.11.549/aws-java-sdk-inspector-1.11.549.jar 2019-12-11T16:46:39.933 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-api-gateway/1.11.549/aws-java-sdk-api-gateway-1.11.549.jar 2019-12-11T16:46:40.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-gamelift/1.11.549/aws-java-sdk-gamelift-1.11.549.jar 2019-12-11T16:46:40.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dms/1.11.549/aws-java-sdk-dms-1.11.549.jar 2019-12-11T16:46:40.033 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacemeteringservice/1.11.549/aws-java-sdk-marketplacemeteringservice-1.11.549.jar 2019-12-11T16:46:40.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot/1.11.549/aws-java-sdk-iot-1.11.549.jar 2019-12-11T16:46:40.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acm/1.11.549/aws-java-sdk-acm-1.11.549.jar 2019-12-11T16:46:40.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-applicationautoscaling/1.11.549/aws-java-sdk-applicationautoscaling-1.11.549.jar 2019-12-11T16:46:40.034 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-discovery/1.11.549/aws-java-sdk-discovery-1.11.549.jar 2019-12-11T16:46:40.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-snowball/1.11.549/aws-java-sdk-snowball-1.11.549.jar 2019-12-11T16:46:40.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidp/1.11.549/aws-java-sdk-cognitoidp-1.11.549.jar 2019-12-11T16:46:40.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-polly/1.11.549/aws-java-sdk-polly-1.11.549.jar 2019-12-11T16:46:40.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rekognition/1.11.549/aws-java-sdk-rekognition-1.11.549.jar 2019-12-11T16:46:40.035 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-stepfunctions/1.11.549/aws-java-sdk-stepfunctions-1.11.549.jar 2019-12-11T16:46:40.036 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-health/1.11.549/aws-java-sdk-health-1.11.549.jar 2019-12-11T16:46:40.036 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costandusagereport/1.11.549/aws-java-sdk-costandusagereport-1.11.549.jar 2019-12-11T16:46:40.036 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codebuild/1.11.549/aws-java-sdk-codebuild-1.11.549.jar 2019-12-11T16:46:40.036 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lightsail/1.11.549/aws-java-sdk-lightsail-1.11.549.jar 2019-12-11T16:46:40.037 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appstream/1.11.549/aws-java-sdk-appstream-1.11.549.jar 2019-12-11T16:46:40.037 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-shield/1.11.549/aws-java-sdk-shield-1.11.549.jar 2019-12-11T16:46:40.037 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-batch/1.11.549/aws-java-sdk-batch-1.11.549.jar 2019-12-11T16:46:40.132 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mechanicalturkrequester/1.11.549/aws-java-sdk-mechanicalturkrequester-1.11.549.jar 2019-12-11T16:46:40.132 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lex/1.11.549/aws-java-sdk-lex-1.11.549.jar 2019-12-11T16:46:40.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workdocs/1.11.549/aws-java-sdk-workdocs-1.11.549.jar 2019-12-11T16:46:40.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-organizations/1.11.549/aws-java-sdk-organizations-1.11.549.jar 2019-12-11T16:46:40.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar 2019-12-11T16:46:40.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/testng/testng/6.1.1/testng-6.1.1.jar 2019-12-11T16:46:40.133 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-java/2.44.0/selenium-java-2.44.0.jar 2019-12-11T16:46:40.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.549/aws-java-sdk-core-1.11.549.jar 2019-12-11T16:46:40.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-htmlunit-driver/2.44.0/selenium-htmlunit-driver-2.44.0.jar 2019-12-11T16:46:40.134 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-server/2.44.0/selenium-server-2.44.0.jar 2019-12-11T16:46:40.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-chrome-driver/2.44.0/selenium-chrome-driver-2.44.0.jar 2019-12-11T16:46:40.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-ie-driver/2.44.0/selenium-ie-driver-2.44.0.jar 2019-12-11T16:46:40.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-safari-driver/2.44.0/selenium-safari-driver-2.44.0.jar 2019-12-11T16:46:40.234 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-firefox-driver/2.44.0/selenium-firefox-driver-2.44.0.jar 2019-12-11T16:46:40.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.3.3/httpmime-4.3.3.jar 2019-12-11T16:46:40.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/2.44.0/selenium-remote-driver-2.44.0.jar 2019-12-11T16:46:40.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.jar 2019-12-11T16:46:40.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-support/2.44.0/selenium-support-2.44.0.jar 2019-12-11T16:46:40.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit/2.15/htmlunit-2.15.jar 2019-12-11T16:46:40.235 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar 2019-12-11T16:46:40.239 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar 2019-12-11T16:46:40.240 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.7.2/jackson-databind-2.6.7.2.jar 2019-12-11T16:46:40.240 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar 2019-12-11T16:46:40.240 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/jmespath-java/1.11.549/jmespath-java-1.11.549.jar 2019-12-11T16:46:40.240 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.48/bcpkix-jdk15on-1.48.jar 2019-12-11T16:46:40.240 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar 2019-12-11T16:46:40.241 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/mx4j/mx4j-tools/3.0.1/mx4j-tools-3.0.1.jar 2019-12-11T16:46:40.241 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.48/bcprov-jdk15on-1.48.jar 2019-12-11T16:46:40.331 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-repacked/7.6.1/jetty-repacked-7.6.1.jar 2019-12-11T16:46:40.331 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-models/1.11.549/aws-java-sdk-models-1.11.549.jar 2019-12-11T16:46:40.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-rc-repacked/5/jetty-rc-repacked-5.jar 2019-12-11T16:46:40.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/mortbay/jetty/servlet-api-2.5/6.1.9/servlet-api-2.5-6.1.9.jar 2019-12-11T16:46:40.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar 2019-12-11T16:46:40.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar 2019-12-11T16:46:40.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.8/snakeyaml-1.8.jar 2019-12-11T16:46:40.332 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/junit/junit/3.8.1/junit-3.8.1.jar 2019-12-11T16:46:40.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar 2019-12-11T16:46:40.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar 2019-12-11T16:46:40.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/beust/jcommander/1.12/jcommander-1.12.jar 2019-12-11T16:46:40.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar 2019-12-11T16:46:40.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar 2019-12-11T16:46:40.333 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec-http/4.1.17.Final/netty-codec-http-4.1.17.Final.jar 2019-12-11T16:46:40.334 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/webbitserver/webbit/0.4.14/webbit-0.4.14.jar 2019-12-11T16:46:40.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-transport/4.1.17.Final/netty-transport-4.1.17.Final.jar 2019-12-11T16:46:40.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec/4.1.17.Final/netty-codec-4.1.17.Final.jar 2019-12-11T16:46:40.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.jar 2019-12-11T16:46:40.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.17.Final/netty-handler-4.1.17.Final.jar 2019-12-11T16:46:40.335 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.17.Final/netty-buffer-4.1.17.Final.jar 2019-12-11T16:46:40.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/jna/3.4.0/jna-3.4.0.jar 2019-12-11T16:46:40.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/platform/3.4.0/platform-3.4.0.jar 2019-12-11T16:46:40.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar 2019-12-11T16:46:40.432 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.17.Final/netty-resolver-4.1.17.Final.jar 2019-12-11T16:46:40.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/cglib/cglib-nodep/2.1_3/cglib-nodep-2.1_3.jar 2019-12-11T16:46:40.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-common/4.1.17.Final/netty-common-4.1.17.Final.jar 2019-12-11T16:46:40.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty/3.5.2.Final/netty-3.5.2.Final.jar 2019-12-11T16:46:40.433 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-api/2.44.0/selenium-api-2.44.0.jar 2019-12-11T16:46:40.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.3/gson-2.3.jar 2019-12-11T16:46:40.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xalan/xalan/2.7.1/xalan-2.7.1.jar 2019-12-11T16:46:40.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/com/google/guava/guava/18.0/guava-18.0.jar 2019-12-11T16:46:40.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar 2019-12-11T16:46:40.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit-core-js/2.15/htmlunit-core-js-2.15.jar 2019-12-11T16:46:40.434 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.11.0/xercesImpl-2.11.0.jar 2019-12-11T16:46:40.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar 2019-12-11T16:46:40.435 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/nekohtml/nekohtml/1.9.21/nekohtml-1.9.21.jar 2019-12-11T16:46:40.436 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/cssparser/cssparser/0.9.14/cssparser-0.9.14.jar 2019-12-11T16:46:40.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-websocket/8.1.15.v20140411/jetty-websocket-8.1.15.v20140411.jar 2019-12-11T16:46:40.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xalan/serializer/2.7.1/serializer-2.7.1.jar 2019-12-11T16:46:40.532 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.jar 2019-12-11T16:46:40.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/w3c/css/sac/1.3/sac-1.3.jar 2019-12-11T16:46:40.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/8.1.15.v20140411/jetty-io-8.1.15.v20140411.jar 2019-12-11T16:46:40.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/8.1.15.v20140411/jetty-http-8.1.15.v20140411.jar 2019-12-11T16:46:40.533 INFO:teuthology.orchestra.run.smithi079.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/8.1.15.v20140411/jetty-util-8.1.15.v20140411.jar 2019-12-11T16:46:49.431 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :compileJava 2019-12-11T16:46:49.431 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :processResources 2019-12-11T16:46:49.431 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :classes 2019-12-11T16:46:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:49.832 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :compileTestJava 2019-12-11T16:46:49.832 INFO:teuthology.orchestra.run.smithi079.stdout:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2019-12-11T16:46:49.832 INFO:teuthology.orchestra.run.smithi079.stdout:Note: Recompile with -Xlint:deprecation for details. 2019-12-11T16:46:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:49.833 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :processTestResources NO-SOURCE 2019-12-11T16:46:49.833 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :testClasses 2019-12-11T16:46:57.431 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:57.431 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :test 2019-12-11T16:46:57.431 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:46:57.432 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 STARTED 2019-12-11T16:47:38.929 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:47:38.930 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 PASSED 2019-12-11T16:47:44.029 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:47:44.030 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 STARTED 2019-12-11T16:47:47.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:47:47.130 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 PASSED 2019-12-11T16:47:54.429 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:47:54.430 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 STARTED 2019-12-11T16:48:28.228 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:48:28.229 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 PASSED 2019-12-11T16:48:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:48:34.429 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 STARTED 2019-12-11T16:48:34.828 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:48:34.829 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 PASSED 2019-12-11T16:48:35.328 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:48:35.329 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 STARTED 2019-12-11T16:48:37.929 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:48:37.930 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 PASSED 2019-12-11T16:48:43.128 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:48:43.128 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 STARTED 2019-12-11T16:48:50.231 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:16:37.181 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.231 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:16:37.185 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.271 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:16:37.225 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.272 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:16:37.226 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.294 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:16:37.215 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.306 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:16:37.227 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.431 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:16:37.382 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.441 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:16:37.395 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:48:50.585 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:16:37.506 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:04.528 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:49:04.529 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 PASSED 2019-12-11T16:49:17.228 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:49:17.228 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 STARTED 2019-12-11T16:49:51.161 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:17:38.115 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.198 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:17:38.153 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.207 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:17:38.129 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.208 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:17:38.130 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.215 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:17:38.169 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.217 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:17:38.170 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.239 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:17:38.194 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.251 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:17:38.173 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:49:51.256 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:17:38.210 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:52:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:52:36.840 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 PASSED 2019-12-11T16:52:48.239 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:52:48.240 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 STARTED 2019-12-11T16:52:50.640 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:52:50.640 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 PASSED 2019-12-11T16:52:56.240 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:52:56.240 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 STARTED 2019-12-11T16:53:00.740 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:53:00.740 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 PASSED 2019-12-11T16:53:08.340 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:53:08.341 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 STARTED 2019-12-11T16:53:27.641 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:53:27.642 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 PASSED 2019-12-11T16:53:36.741 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:53:36.742 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 STARTED 2019-12-11T16:53:37.641 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:53:37.643 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 PASSED 2019-12-11T16:53:39.041 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:53:39.042 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 STARTED 2019-12-11T16:53:58.343 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:21:45.283 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.352 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:21:45.292 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.361 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:21:45.301 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.369 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:21:45.309 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.418 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:21:45.340 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.421 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:21:45.344 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.501 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:21:45.442 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.549 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:21:45.489 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:53:58.583 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:21:45.506 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:54:12.082 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T16:54:12.015+0000 41264700 -1 handle_data failed with exit code -5 2019-12-11T16:54:12.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:54:12.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 PASSED 2019-12-11T16:54:18.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:54:18.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 STARTED 2019-12-11T16:54:43.147 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:54:43.147 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 PASSED 2019-12-11T16:54:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:54:54.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 STARTED 2019-12-11T16:55:15.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:55:15.145 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 PASSED 2019-12-11T16:55:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T16:55:23.145 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 STARTED 2019-12-11T16:58:12.578 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:25:59.513 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.583 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:25:59.518 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.622 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:25:59.556 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.628 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:25:59.562 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.658 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:25:59.568 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.659 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:25:59.570 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.723 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:25:59.657 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.764 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:25:59.698 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T16:58:12.780 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:25:59.692 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:02:56.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:02:56.476 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 PASSED 2019-12-11T17:03:05.150 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:03:05.151 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 STARTED 2019-12-11T17:03:45.651 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:03:45.651 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 PASSED 2019-12-11T17:03:52.451 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:03:52.451 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 STARTED 2019-12-11T17:04:20.643 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:32:07.573 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.646 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:32:07.577 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.661 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:32:07.566 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.670 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:32:07.574 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.690 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:32:07.621 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.704 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:32:07.628 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.877 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:32:07.809 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.908 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:32:07.812 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:20.941 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:32:07.872 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:04:45.051 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:04:45.051 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 PASSED 2019-12-11T17:04:53.751 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:04:53.751 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 STARTED 2019-12-11T17:05:01.751 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:01.751 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 PASSED 2019-12-11T17:05:15.351 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:15.351 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 STARTED 2019-12-11T17:05:16.751 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:16.751 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 PASSED 2019-12-11T17:05:19.851 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:19.851 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 STARTED 2019-12-11T17:05:21.551 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:21.551 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 PASSED 2019-12-11T17:05:28.452 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:28.452 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 STARTED 2019-12-11T17:05:30.651 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:30.651 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 PASSED 2019-12-11T17:05:35.151 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:35.151 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 STARTED 2019-12-11T17:05:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:38.352 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 PASSED 2019-12-11T17:05:44.551 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:44.551 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 STARTED 2019-12-11T17:05:47.351 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:47.351 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 PASSED 2019-12-11T17:05:53.751 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:53.751 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 STARTED 2019-12-11T17:05:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:05:57.252 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 PASSED 2019-12-11T17:06:04.451 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:06:04.451 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 STARTED 2019-12-11T17:06:05.951 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:06:05.952 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 PASSED 2019-12-11T17:06:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:06:11.251 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 STARTED 2019-12-11T17:06:46.051 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:06:46.051 INFO:teuthology.orchestra.run.smithi079.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 PASSED 2019-12-11T17:06:53.751 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:06:53.752 INFO:teuthology.orchestra.run.smithi079.stdout:---------------------------------------------------------------------- 2019-12-11T17:06:53.752 INFO:teuthology.orchestra.run.smithi079.stdout:| Results: SUCCESS (26 tests, 26 successes, 0 failures, 0 skipped) | 2019-12-11T17:06:53.752 INFO:teuthology.orchestra.run.smithi079.stdout:---------------------------------------------------------------------- 2019-12-11T17:06:53.752 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:06:53.753 INFO:teuthology.orchestra.run.smithi079.stdout:BUILD SUCCESSFUL in 20m 22s 2019-12-11T17:06:53.753 INFO:teuthology.orchestra.run.smithi079.stdout:5 actionable tasks: 4 executed, 1 up-to-date 2019-12-11T17:06:54.081 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:06:54.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.198+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.224 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.220+0000 7f8293948ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:06:54.224 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.220+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.235+0000 7f8293948ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:06:54.239 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.235+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.242 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.240+0000 7f8293948ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:06:54.242 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.240+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.250 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.247+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:06:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.406+0000 7f8293948ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:06:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.406+0000 7f8293948ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:06:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.406+0000 7f8293948ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:06:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.406+0000 7f8293948ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:06:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.406+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.423 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.420+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:06:54.423 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.420+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.436 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.433+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:06:54.436 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.433+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.446 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.443+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:06:54.446 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.443+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.457+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:06:54.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.457+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.466 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.464+0000 7f8293948ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:06:54.466 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.464+0000 7f8293948ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:06:54.467 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.464+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.469+0000 7f8293948ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:06:54.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.469+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.480 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.478+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:06:54.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.478+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.494 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.491+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:06:54.494 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.491+0000 7f8293948ec0 20 zonegroup default 2019-12-11T17:06:54.494 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.491+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.497+0000 7f8293948ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:06:54.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.497+0000 7f8293948ec0 10 Cannot find current period zone using local zone 2019-12-11T17:06:54.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.497+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.503+0000 7f8293948ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:06:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.503+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.515 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.510+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:06:54.515 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.510+0000 7f8293948ec0 20 rados->read ofs=0 len=0 2019-12-11T17:06:54.524 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.521+0000 7f8293948ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:06:54.524 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.521+0000 7f8293948ec0 20 zone default found 2019-12-11T17:06:54.524 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.521+0000 7f8293948ec0 20 started sync module instance, tier type = 2019-12-11T17:06:54.524 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.521+0000 7f8293948ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:06:54.880 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.877+0000 7f8293948ec0 20 add_watcher() i=0 2019-12-11T17:06:54.880 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.877+0000 7f8293948ec0 20 add_watcher() i=1 2019-12-11T17:06:54.880 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.877+0000 7f8293948ec0 20 add_watcher() i=2 2019-12-11T17:06:54.880 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.877+0000 7f8293948ec0 20 add_watcher() i=3 2019-12-11T17:06:54.880 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.877+0000 7f8293948ec0 20 add_watcher() i=4 2019-12-11T17:06:54.881 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.877+0000 7f8293948ec0 20 add_watcher() i=5 2019-12-11T17:06:54.892 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.889+0000 7f8293948ec0 20 add_watcher() i=6 2019-12-11T17:06:54.929 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.924+0000 7f8293948ec0 20 add_watcher() i=7 2019-12-11T17:06:54.929 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.924+0000 7f8293948ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:06:54.938 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.924+0000 7f8250fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:06:54.939 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.935+0000 7f8281373700 0 -- 172.21.15.79:0/4104278464 >> v2:172.21.15.199:3300/0 conn(0x5595554b9230 msgr2=0x5595554cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:06:54.986 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:54.983+0000 7f8281373700 0 -- 172.21.15.79:0/2365651272 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x5595554e5020 msgr2=0x5595554c81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:06:56.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.253+0000 7f81c2ffd700 20 reqs_thread_entry: start 2019-12-11T17:06:56.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.253+0000 7f81c0ff9700 20 reqs_thread_entry: start 2019-12-11T17:06:56.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.254+0000 7f8293948ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:06:56.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.254+0000 7f81b6ffd700 20 reqs_thread_entry: start 2019-12-11T17:06:56.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.254+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:06:56.297 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.295+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:56.322 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.319+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:56.345 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:06:56.345 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.415', time=2019-12-11T18:49:07.029101+0000, chain.objs.size()=9 2019-12-11T17:06:56.345 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.1 2019-12-11T17:06:56.345 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.1_1 2019-12-11T17:06:56.345 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.2 2019-12-11T17:06:56.345 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.2_1 2019-12-11T17:06:56.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.3 2019-12-11T17:06:56.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.3_1 2019-12-11T17:06:56.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.4 2019-12-11T17:06:56.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.4_1 2019-12-11T17:06:56.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.5 2019-12-11T17:06:56.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.472', time=2019-12-11T18:52:38.338948+0000, chain.objs.size()=7 2019-12-11T17:06:56.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~W8yxDdBQKO0XRPPfI4iE8NJcg-AZ44_.1 2019-12-11T17:06:56.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~W8yxDdBQKO0XRPPfI4iE8NJcg-AZ44_.1_1 2019-12-11T17:06:56.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.341+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~W8yxDdBQKO0XRPPfI4iE8NJcg-AZ44_.1_2 2019-12-11T17:06:56.627 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.625+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~W8yxDdBQKO0XRPPfI4iE8NJcg-AZ44_.2 2019-12-11T17:06:56.628 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.625+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~W8yxDdBQKO0XRPPfI4iE8NJcg-AZ44_.2_1 2019-12-11T17:06:56.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.660+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~W8yxDdBQKO0XRPPfI4iE8NJcg-AZ44_.2_2 2019-12-11T17:06:56.663 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.660+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~W8yxDdBQKO0XRPPfI4iE8NJcg-AZ44_.3 2019-12-11T17:06:56.819 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.816+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:06:56.954 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.951+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:06:56.996 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:56.994+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.013+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:57.028 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.026+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.064+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:06:57.133 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.130+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.153 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.150+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:57.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.160+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.191+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:06:57.232 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.229+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.245+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:57.259 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.256+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.288+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:06:57.378 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.376+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.410 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.408+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:57.441 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.438+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:06:57.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.438+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.460', time=2019-12-11T18:52:13.549874+0000, chain.objs.size()=7 2019-12-11T17:06:57.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.439+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~WU-e9cg1VvxK8e0Q6pOY9gvjJSXeKw_.1 2019-12-11T17:06:57.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.439+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~WU-e9cg1VvxK8e0Q6pOY9gvjJSXeKw_.1_1 2019-12-11T17:06:57.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.439+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~WU-e9cg1VvxK8e0Q6pOY9gvjJSXeKw_.1_2 2019-12-11T17:06:57.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.439+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~WU-e9cg1VvxK8e0Q6pOY9gvjJSXeKw_.2 2019-12-11T17:06:57.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.439+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~WU-e9cg1VvxK8e0Q6pOY9gvjJSXeKw_.2_1 2019-12-11T17:06:57.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.439+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~WU-e9cg1VvxK8e0Q6pOY9gvjJSXeKw_.2_2 2019-12-11T17:06:57.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.439+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~WU-e9cg1VvxK8e0Q6pOY9gvjJSXeKw_.3 2019-12-11T17:06:57.783 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.780+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:06:57.891 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.888+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:06:57.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.941+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:57.959 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.956+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:57.980 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:06:57.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.628', time=2019-12-11T19:01:54.623628+0000, chain.objs.size()=16 2019-12-11T17:06:57.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.1 2019-12-11T17:06:57.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.1_1 2019-12-11T17:06:57.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.1_2 2019-12-11T17:06:57.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.2 2019-12-11T17:06:57.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.2_1 2019-12-11T17:06:57.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.2_2 2019-12-11T17:06:57.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.3 2019-12-11T17:06:57.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.3_1 2019-12-11T17:06:57.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.3_2 2019-12-11T17:06:57.983 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.4 2019-12-11T17:06:57.984 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.4_1 2019-12-11T17:06:57.984 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:57.977+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.4_2 2019-12-11T17:06:58.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.181+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.5 2019-12-11T17:06:58.222 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.220+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.5_1 2019-12-11T17:06:58.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.5_2 2019-12-11T17:06:58.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.238+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~5IAky97ozjInQwZaxVqOQcHtQOJ0Sq3.6 2019-12-11T17:06:58.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.574+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:06:58.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.666+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:06:58.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.722+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.742+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:58.756 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.754+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:58.797 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.795+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:06:58.869 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.867+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:58.895 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.893+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:58.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:06:58.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.386', time=2019-12-11T18:48:29.876065+0000, chain.objs.size()=9 2019-12-11T17:06:58.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__multipart_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.1 2019-12-11T17:06:58.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__shadow_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.1_1 2019-12-11T17:06:58.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__multipart_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.2 2019-12-11T17:06:58.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__shadow_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.2_1 2019-12-11T17:06:58.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__multipart_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.3 2019-12-11T17:06:58.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__shadow_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.3_1 2019-12-11T17:06:58.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__multipart_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.4 2019-12-11T17:06:58.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__shadow_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.4_1 2019-12-11T17:06:58.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:58.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.15__multipart_key1.2~H9IYbpw-2LeJbpWK37s-gOQszWmoZ0y.5 2019-12-11T17:06:59.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.196+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:06:59.298 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.295+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:06:59.369 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.366+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:59.388 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.385+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:59.403 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.401+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:59.436 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.433+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:06:59.499 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.496+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:59.527 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.524+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:59.540 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.538+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:59.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.591+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:06:59.656 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.653+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:59.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.683+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:59.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.696+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:59.751 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.749+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:06:59.798 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.795+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:06:59.814 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.811+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:06:59.835 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:06:59.836 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.528', time=2019-12-11T18:54:15.392472+0000, chain.objs.size()=9 2019-12-11T17:06:59.836 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__multipart_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.1 2019-12-11T17:06:59.836 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__shadow_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.1_1 2019-12-11T17:06:59.836 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__multipart_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.2 2019-12-11T17:06:59.836 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__shadow_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.2_1 2019-12-11T17:06:59.836 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__multipart_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.3 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__shadow_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.3_1 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__multipart_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.4 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__shadow_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.4_1 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.25__multipart_key1.2~K35O3bldHRG3kJuGCj5uhXVy_CvAlvd.5 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.636', time=2019-12-11T19:02:55.199454+0000, chain.objs.size()=16 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.1 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.1_1 2019-12-11T17:06:59.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:06:59.832+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.1_2 2019-12-11T17:07:00.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.2 2019-12-11T17:07:00.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.2_1 2019-12-11T17:07:00.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.2_2 2019-12-11T17:07:00.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.3 2019-12-11T17:07:00.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.3_1 2019-12-11T17:07:00.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.3_2 2019-12-11T17:07:00.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.4 2019-12-11T17:07:00.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.4_1 2019-12-11T17:07:00.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.193+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.4_2 2019-12-11T17:07:00.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.244+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.5 2019-12-11T17:07:00.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.245+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.5_1 2019-12-11T17:07:00.540 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.537+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.5_2 2019-12-11T17:07:00.540 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.537+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~Uj0QJZ2sW8bDjF9txuaNf9kWbykdLkb.6 2019-12-11T17:07:00.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.754+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:00.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.840+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:07:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.874+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:00.892 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.890+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:00.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.898+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:00.931 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.928+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:07:00.974 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.972+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:00.989 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:00.987+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:01.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:01.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK', time=2019-12-11T18:47:38.098157+0000, chain.objs.size()=9 2019-12-11T17:07:01.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__multipart_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.1 2019-12-11T17:07:01.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__shadow_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.1_1 2019-12-11T17:07:01.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__multipart_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.2 2019-12-11T17:07:01.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__shadow_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.2_1 2019-12-11T17:07:01.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__multipart_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.3 2019-12-11T17:07:01.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__shadow_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.3_1 2019-12-11T17:07:01.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__multipart_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.4 2019-12-11T17:07:01.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__shadow_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.4_1 2019-12-11T17:07:01.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.000+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.13__multipart_key1.2~aRDIdNHbRfgr6ItoSldQPT2v-6R9CfK.5 2019-12-11T17:07:01.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.273+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:01.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.425+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:07:01.507 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.504+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:01.535 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.533+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:01.547 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.545+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:01.602 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.598+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:07:01.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.687+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:01.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.704+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:01.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:07:01.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.620', time=2019-12-11T19:00:51.294896+0000, chain.objs.size()=19 2019-12-11T17:07:01.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.1 2019-12-11T17:07:01.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.1_1 2019-12-11T17:07:01.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.2 2019-12-11T17:07:01.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.2_1 2019-12-11T17:07:01.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.3 2019-12-11T17:07:01.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.3_1 2019-12-11T17:07:01.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.4 2019-12-11T17:07:01.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.4_1 2019-12-11T17:07:01.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.5 2019-12-11T17:07:01.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.5_1 2019-12-11T17:07:01.747 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.6 2019-12-11T17:07:01.747 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:01.742+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.6_1 2019-12-11T17:07:02.070 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.7 2019-12-11T17:07:02.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.7_1 2019-12-11T17:07:02.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.8 2019-12-11T17:07:02.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.8_1 2019-12-11T17:07:02.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.9 2019-12-11T17:07:02.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.9_1 2019-12-11T17:07:02.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~2YAKVKhY7EfHsc8gsl1hnSIEPpjaGMI.10 2019-12-11T17:07:02.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.837', time=2019-12-11T19:06:49.155216+0000, chain.objs.size()=21 2019-12-11T17:07:02.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.1 2019-12-11T17:07:02.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.1_1 2019-12-11T17:07:02.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.2 2019-12-11T17:07:02.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.067+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.2_1 2019-12-11T17:07:02.236 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.233+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.3 2019-12-11T17:07:02.331 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.328+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.3_1 2019-12-11T17:07:02.331 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.328+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.4 2019-12-11T17:07:02.332 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.328+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.4_1 2019-12-11T17:07:02.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.354+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.5 2019-12-11T17:07:02.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.354+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.5_1 2019-12-11T17:07:02.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.354+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.6 2019-12-11T17:07:02.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.354+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.6_1 2019-12-11T17:07:02.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.576+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.7 2019-12-11T17:07:02.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.576+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.7_1 2019-12-11T17:07:02.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.576+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.8 2019-12-11T17:07:02.691 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.689+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.8_1 2019-12-11T17:07:02.692 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.689+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.9 2019-12-11T17:07:02.875 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.871+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.9_1 2019-12-11T17:07:02.875 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.871+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.10 2019-12-11T17:07:02.875 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.871+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__shadow_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.10_1 2019-12-11T17:07:02.876 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:02.871+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.39__multipart_key1.2~nAo8Lkpl_hL0AK0riWDqOjytDg5AcES.11 2019-12-11T17:07:03.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.279+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:03.421 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.419+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:07:03.478 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.475+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:03.493 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.491+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:03.503 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.500+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:03.545 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.542+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:07:03.593 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.590+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:03.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.606+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:03.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:07:03.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.595', time=2019-12-11T18:58:12.929283+0000, chain.objs.size()=21 2019-12-11T17:07:03.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.1 2019-12-11T17:07:03.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.1_1 2019-12-11T17:07:03.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.2 2019-12-11T17:07:03.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.2_1 2019-12-11T17:07:03.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.3 2019-12-11T17:07:03.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.3_1 2019-12-11T17:07:03.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.4 2019-12-11T17:07:03.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.4_1 2019-12-11T17:07:03.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.5 2019-12-11T17:07:03.635 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.5_1 2019-12-11T17:07:03.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.6 2019-12-11T17:07:03.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.631+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.6_1 2019-12-11T17:07:03.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.7 2019-12-11T17:07:03.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.7_1 2019-12-11T17:07:03.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.8 2019-12-11T17:07:03.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.8_1 2019-12-11T17:07:03.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.9 2019-12-11T17:07:03.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.9_1 2019-12-11T17:07:03.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.10 2019-12-11T17:07:03.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.10_1 2019-12-11T17:07:03.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.917+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~8cudA1pCBTxIXoPqbcv0OS_lSdaa5Hq.11 2019-12-11T17:07:03.954 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.951+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.644', time=2019-12-11T19:02:59.146008+0000, chain.objs.size()=16 2019-12-11T17:07:03.954 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.951+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.1 2019-12-11T17:07:03.954 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:03.951+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.1_1 2019-12-11T17:07:04.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.128+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.1_2 2019-12-11T17:07:04.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.183+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.2 2019-12-11T17:07:04.239 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.2_1 2019-12-11T17:07:04.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.2_2 2019-12-11T17:07:04.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.3 2019-12-11T17:07:04.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.3_1 2019-12-11T17:07:04.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.3_2 2019-12-11T17:07:04.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.4 2019-12-11T17:07:04.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.4_1 2019-12-11T17:07:04.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.237+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.4_2 2019-12-11T17:07:04.245 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.5 2019-12-11T17:07:04.650 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.648+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.5_1 2019-12-11T17:07:04.651 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.648+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.5_2 2019-12-11T17:07:04.651 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.648+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~oF0PWSVq3IAC0YDCJxNqcxJtyQs8qk1.6 2019-12-11T17:07:04.869 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.862+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:04.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:04.976+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:07:05.022 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.020+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.050+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.438', time=2019-12-11T18:50:37.276308+0000, chain.objs.size()=9 2019-12-11T17:07:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.1 2019-12-11T17:07:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.1_1 2019-12-11T17:07:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.2 2019-12-11T17:07:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.2_1 2019-12-11T17:07:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.3 2019-12-11T17:07:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.3_1 2019-12-11T17:07:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.4 2019-12-11T17:07:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.4_1 2019-12-11T17:07:05.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.064+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~nwI_IXkyhi0Wvuhednn_qtnzSbThUp_.5 2019-12-11T17:07:05.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.376+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.463+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:07:05.506 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.502+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:05.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.517+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.525+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:05.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.561+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:07:05.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.644+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:05.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.672+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:05.709 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:07:05.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.509', time=2019-12-11T18:53:30.289915+0000, chain.objs.size()=9 2019-12-11T17:07:05.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__multipart_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.1 2019-12-11T17:07:05.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__shadow_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.1_1 2019-12-11T17:07:05.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__multipart_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.2 2019-12-11T17:07:05.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__shadow_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.2_1 2019-12-11T17:07:05.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__multipart_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.3 2019-12-11T17:07:05.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__shadow_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.3_1 2019-12-11T17:07:05.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__multipart_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.4 2019-12-11T17:07:05.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__shadow_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.4_1 2019-12-11T17:07:05.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.24__multipart_key1.2~IWUiu_Ygfnm98cx1T0BkoitA8vDqBR0.5 2019-12-11T17:07:05.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.674', time=2019-12-11T19:04:47.497636+0000, chain.objs.size()=9 2019-12-11T17:07:05.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__multipart_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.1 2019-12-11T17:07:05.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__shadow_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.1_1 2019-12-11T17:07:05.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.706+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__multipart_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.2 2019-12-11T17:07:05.891 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:05.888+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__shadow_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.2_1 2019-12-11T17:07:06.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.166+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__multipart_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.3 2019-12-11T17:07:06.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.166+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__shadow_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.3_1 2019-12-11T17:07:06.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.166+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__multipart_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.4 2019-12-11T17:07:06.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.166+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__shadow_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.4_1 2019-12-11T17:07:06.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.166+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.30__multipart_key1.2~Rnd0cEHtGQeH4ABNWmzma712aYoYzoM.5 2019-12-11T17:07:06.382 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.379+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:06.462 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.460+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:07:06.507 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.504+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:06.524 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.521+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:06.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.529+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:06.580 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.577+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:07:06.631 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.628+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:06.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.645+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:06.684 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:07:06.684 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~49LPaDQFVn--BC8M6dydESf-QRoh6_z', time=2019-12-11T18:55:19.085161+0000, chain.objs.size()=5 2019-12-11T17:07:06.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.27__multipart_key1.2~49LPaDQFVn--BC8M6dydESf-QRoh6_z.7 2019-12-11T17:07:06.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.27__shadow_key1.2~49LPaDQFVn--BC8M6dydESf-QRoh6_z.7_1 2019-12-11T17:07:06.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.27__multipart_key1.2~49LPaDQFVn--BC8M6dydESf-QRoh6_z.10 2019-12-11T17:07:06.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.27__shadow_key1.2~49LPaDQFVn--BC8M6dydESf-QRoh6_z.10_1 2019-12-11T17:07:06.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.27__multipart_key1.2~49LPaDQFVn--BC8M6dydESf-QRoh6_z.13 2019-12-11T17:07:06.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0', time=2019-12-11T19:05:11.344332+0000, chain.objs.size()=38 2019-12-11T17:07:06.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.1 2019-12-11T17:07:06.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.2 2019-12-11T17:07:06.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.3 2019-12-11T17:07:06.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.4 2019-12-11T17:07:06.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.5 2019-12-11T17:07:06.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.6 2019-12-11T17:07:06.687 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.681+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.7 2019-12-11T17:07:06.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.910+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.8 2019-12-11T17:07:06.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.911+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.9 2019-12-11T17:07:06.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.911+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.10 2019-12-11T17:07:06.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.911+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.11 2019-12-11T17:07:06.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.911+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.12 2019-12-11T17:07:06.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.911+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.13 2019-12-11T17:07:06.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.911+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.14 2019-12-11T17:07:06.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.911+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.Z1AxXGiYzNDQEAutRpfcofVrRXggN96.15 2019-12-11T17:07:06.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:06.912+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.K2NOXX--cmwO4Z8Y3jmGx8vXFY2m0oA.16 2019-12-11T17:07:07.032 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.029+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.17 2019-12-11T17:07:07.032 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.029+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.7MQ0u0o_SoVvkeeEDlgvei_5i_ElU2P.18 2019-12-11T17:07:07.052 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.050+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.A1dQ4Y_Mb1ovuIyRTU51kCbricVVJhc.19 2019-12-11T17:07:07.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.dfzR3FFtJGiZvZG9CJ052fsnT24LgS4.20 2019-12-11T17:07:07.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.21 2019-12-11T17:07:07.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.22 2019-12-11T17:07:07.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.23 2019-12-11T17:07:07.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.24 2019-12-11T17:07:07.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.25 2019-12-11T17:07:07.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.26 2019-12-11T17:07:07.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.243+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.27 2019-12-11T17:07:07.361 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.358+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.28 2019-12-11T17:07:07.361 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.358+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.29 2019-12-11T17:07:07.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.358+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.30 2019-12-11T17:07:07.537 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.534+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.31 2019-12-11T17:07:07.591 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.588+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.32 2019-12-11T17:07:07.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.600+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.33 2019-12-11T17:07:07.623 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.621+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.34 2019-12-11T17:07:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.621+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.35 2019-12-11T17:07:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.621+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.36 2019-12-11T17:07:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.621+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.37 2019-12-11T17:07:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.621+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.31__multipart_key1.2~s3Y_gIp472F4E0WsN6gYO7bSvHcQnD0.38 2019-12-11T17:07:07.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:07.900+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:08.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.012+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:07:08.060 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.057+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:08.078 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.076+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:08.092 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.089+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.119+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:07:08.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.163+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:08.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.181+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:08.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:08.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.446', time=2019-12-11T18:51:08.928734+0000, chain.objs.size()=9 2019-12-11T17:07:08.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.1 2019-12-11T17:07:08.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.1_1 2019-12-11T17:07:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.2 2019-12-11T17:07:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.2_1 2019-12-11T17:07:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.3 2019-12-11T17:07:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.3_1 2019-12-11T17:07:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.4 2019-12-11T17:07:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.4_1 2019-12-11T17:07:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.194+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~AY4S9QUdWTbVEW3kN8jktsrd00f1yNR.5 2019-12-11T17:07:08.432 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.429+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:08.587 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.584+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:07:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.625+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:08.646 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.643+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:08.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:08.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.454', time=2019-12-11T18:51:50.410381+0000, chain.objs.size()=9 2019-12-11T17:07:08.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.1 2019-12-11T17:07:08.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.1_1 2019-12-11T17:07:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.2 2019-12-11T17:07:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.2_1 2019-12-11T17:07:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.3 2019-12-11T17:07:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.3_1 2019-12-11T17:07:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.4 2019-12-11T17:07:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.4_1 2019-12-11T17:07:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.662+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~1fm9b-T2OwsfOyvVAqjPR_4DssKRjYo.5 2019-12-11T17:07:08.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:08.969+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:09.060 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.057+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:07:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.096+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:09.114 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.112+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:09.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:09.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.466', time=2019-12-11T18:52:35.598724+0000, chain.objs.size()=7 2019-12-11T17:07:09.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~lgbitbcTfKvnoaNhPisZNzlSkLX1gey.1 2019-12-11T17:07:09.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~lgbitbcTfKvnoaNhPisZNzlSkLX1gey.1_1 2019-12-11T17:07:09.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~lgbitbcTfKvnoaNhPisZNzlSkLX1gey.1_2 2019-12-11T17:07:09.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~lgbitbcTfKvnoaNhPisZNzlSkLX1gey.2 2019-12-11T17:07:09.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~lgbitbcTfKvnoaNhPisZNzlSkLX1gey.2_1 2019-12-11T17:07:09.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__shadow_key1.2~lgbitbcTfKvnoaNhPisZNzlSkLX1gey.2_2 2019-12-11T17:07:09.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.153+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.20__multipart_key1.2~lgbitbcTfKvnoaNhPisZNzlSkLX1gey.3 2019-12-11T17:07:09.305 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.302+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.492+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:07:09.534 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.531+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:09.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.546+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:09.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:09.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.417', time=2019-12-11T18:49:13.473260+0000, chain.objs.size()=9 2019-12-11T17:07:09.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.1 2019-12-11T17:07:09.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.1_1 2019-12-11T17:07:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.2 2019-12-11T17:07:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.2_1 2019-12-11T17:07:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.3 2019-12-11T17:07:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.3_1 2019-12-11T17:07:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.4 2019-12-11T17:07:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__shadow_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.4_1 2019-12-11T17:07:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.559+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.17__multipart_key1.2~U1JWogf1JKhccc6X8lTbyyqxEf9ay1y.5 2019-12-11T17:07:09.840 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.837+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:09.939 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.936+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:07:09.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.977+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:09.994 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:09.991+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:10.006 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.003+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:10.006 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.003+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.430', time=2019-12-11T18:52:44.106194+0000, chain.objs.size()=5 2019-12-11T17:07:10.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.003+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.19__shadow_.4EzlMNKY82sg-tGaIbvjCOr-94FAU4G_1 2019-12-11T17:07:10.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.003+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.19__shadow_.4EzlMNKY82sg-tGaIbvjCOr-94FAU4G_2 2019-12-11T17:07:10.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.003+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.19__shadow_.4EzlMNKY82sg-tGaIbvjCOr-94FAU4G_3 2019-12-11T17:07:10.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.003+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.19__shadow_.4EzlMNKY82sg-tGaIbvjCOr-94FAU4G_4 2019-12-11T17:07:10.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.003+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.19__shadow_.4EzlMNKY82sg-tGaIbvjCOr-94FAU4G_5 2019-12-11T17:07:10.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.169+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:10.391 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.388+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:07:10.425 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.423+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:10.440 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.437+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:10.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:07:10.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.659', time=2019-12-11T19:03:47.523047+0000, chain.objs.size()=9 2019-12-11T17:07:10.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__multipart_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.1 2019-12-11T17:07:10.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__shadow_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.1_1 2019-12-11T17:07:10.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__multipart_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.2 2019-12-11T17:07:10.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__shadow_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.2_1 2019-12-11T17:07:10.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__multipart_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.3 2019-12-11T17:07:10.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__shadow_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.3_1 2019-12-11T17:07:10.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__multipart_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.4 2019-12-11T17:07:10.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__shadow_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.4_1 2019-12-11T17:07:10.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.450+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.29__multipart_key1.2~vhsyAekGj2umsc9vCLu6Ge4XzW4N2_5.5 2019-12-11T17:07:10.765 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.762+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:10.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.903+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:07:10.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.960+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:10.977 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.975+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:11.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2019-12-11T17:07:11.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.560', time=2019-12-11T18:54:44.776714+0000, chain.objs.size()=9 2019-12-11T17:07:11.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.1 2019-12-11T17:07:11.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.1_1 2019-12-11T17:07:11.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.2 2019-12-11T17:07:11.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.2_1 2019-12-11T17:07:11.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.3 2019-12-11T17:07:11.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.3_1 2019-12-11T17:07:11.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.4 2019-12-11T17:07:11.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.4_1 2019-12-11T17:07:11.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file.mpg.2~GkaDyysRXIsEvDtP_SGP0eObWr-v-59.5 2019-12-11T17:07:11.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.559', time=2019-12-11T18:54:46.824291+0000, chain.objs.size()=9 2019-12-11T17:07:11.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.1 2019-12-11T17:07:11.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.1_1 2019-12-11T17:07:11.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:10.999+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.2 2019-12-11T17:07:11.302 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.300+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.2_1 2019-12-11T17:07:11.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.300+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.3 2019-12-11T17:07:11.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.3_1 2019-12-11T17:07:11.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.4 2019-12-11T17:07:11.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__shadow_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.4_1 2019-12-11T17:07:11.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.26__multipart_file2.mpg.2~pBnVz8OPaM_lFxcaZYp_2YwQftzETE-.5 2019-12-11T17:07:11.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.608', time=2019-12-11T18:59:37.705615+0000, chain.objs.size()=21 2019-12-11T17:07:11.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.1 2019-12-11T17:07:11.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.1_1 2019-12-11T17:07:11.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.2 2019-12-11T17:07:11.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.2_1 2019-12-11T17:07:11.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.392+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.3 2019-12-11T17:07:11.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.505+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.3_1 2019-12-11T17:07:11.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.505+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.4 2019-12-11T17:07:11.649 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.644+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.4_1 2019-12-11T17:07:11.649 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.646+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.5 2019-12-11T17:07:11.657 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.654+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.5_1 2019-12-11T17:07:11.657 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.654+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.6 2019-12-11T17:07:11.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.672+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.6_1 2019-12-11T17:07:11.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.672+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.7 2019-12-11T17:07:11.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.800+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.7_1 2019-12-11T17:07:11.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.800+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.8 2019-12-11T17:07:11.804 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.800+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.8_1 2019-12-11T17:07:11.817 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.814+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.9 2019-12-11T17:07:11.817 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:11.814+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.9_1 2019-12-11T17:07:12.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.023+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.10 2019-12-11T17:07:12.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.023+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__shadow_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.10_1 2019-12-11T17:07:12.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.066+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.28__multipart_key1.2~G5SfYNEq6gNPfH1JygQ6cFepVVn3vBW.11 2019-12-11T17:07:12.291 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.288+0000 7f8293948ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:07:12.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.405+0000 7f8293948ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:07:12.461 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.453+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:12.482 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.476+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:12.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.485+0000 7f8293948ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:12.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.563+0000 7f8293948ec0 20 remove_watcher() i=0 2019-12-11T17:07:12.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.563+0000 7f8293948ec0 2 removed watcher, disabling cache 2019-12-11T17:07:12.599 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.596+0000 7f8293948ec0 20 remove_watcher() i=1 2019-12-11T17:07:12.644 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.642+0000 7f8293948ec0 20 remove_watcher() i=2 2019-12-11T17:07:12.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.684+0000 7f8293948ec0 20 remove_watcher() i=3 2019-12-11T17:07:12.723 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.720+0000 7f8293948ec0 20 remove_watcher() i=4 2019-12-11T17:07:12.756 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.754+0000 7f8293948ec0 20 remove_watcher() i=5 2019-12-11T17:07:12.796 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.793+0000 7f8293948ec0 20 remove_watcher() i=6 2019-12-11T17:07:12.833 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.831+0000 7f8293948ec0 20 remove_watcher() i=7 2019-12-11T17:07:12.840 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:07:12.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.945+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:12.969 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.966+0000 7fac8db1eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:12.969 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.966+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:12.978 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.974+0000 7fac8db1eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:12.978 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.974+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:12.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.979+0000 7fac8db1eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:12.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.979+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:12.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:12.989+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:13.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.180+0000 7fac8db1eec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:07:13.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.180+0000 7fac8db1eec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:07:13.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.180+0000 7fac8db1eec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:07:13.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.180+0000 7fac8db1eec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:07:13.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.180+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.189+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:13.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.189+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.207+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:07:13.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.207+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.219+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.219+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.234+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.234+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.243 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.241+0000 7fac8db1eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:13.244 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.241+0000 7fac8db1eec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:07:13.244 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.241+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.250 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.248+0000 7fac8db1eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:13.250 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.248+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.256 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.254+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:13.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.254+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.273+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:07:13.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.273+0000 7fac8db1eec0 20 zonegroup default 2019-12-11T17:07:13.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.273+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.280+0000 7fac8db1eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:13.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.280+0000 7fac8db1eec0 10 Cannot find current period zone using local zone 2019-12-11T17:07:13.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.280+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.287 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.285+0000 7fac8db1eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:13.288 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.285+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.297 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.295+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:13.298 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.295+0000 7fac8db1eec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:13.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.313+0000 7fac8db1eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:07:13.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.313+0000 7fac8db1eec0 20 zone default found 2019-12-11T17:07:13.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.313+0000 7fac8db1eec0 20 started sync module instance, tier type = 2019-12-11T17:07:13.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.313+0000 7fac8db1eec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:07:13.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.724+0000 7fac8db1eec0 20 add_watcher() i=0 2019-12-11T17:07:13.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.724+0000 7fac8db1eec0 20 add_watcher() i=1 2019-12-11T17:07:13.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.724+0000 7fac8db1eec0 20 add_watcher() i=2 2019-12-11T17:07:13.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.724+0000 7fac8db1eec0 20 add_watcher() i=3 2019-12-11T17:07:13.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.724+0000 7fac8db1eec0 20 add_watcher() i=4 2019-12-11T17:07:13.728 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.724+0000 7fac8db1eec0 20 add_watcher() i=5 2019-12-11T17:07:13.728 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.724+0000 7fac8db1eec0 20 add_watcher() i=6 2019-12-11T17:07:13.761 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.758+0000 7fac8db1eec0 20 add_watcher() i=7 2019-12-11T17:07:13.762 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.758+0000 7fac8db1eec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:07:13.762 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.759+0000 7fac4a7ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:07:13.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.764+0000 7fac73fff700 0 -- 172.21.15.79:0/1389444545 >> v1:172.21.15.79:6789/0 conn(0x559f17c6cea0 legacy=0x559f17c69a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:07:13.806 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:13.803+0000 7fac73fff700 0 -- 172.21.15.79:0/636532300 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x559f17cb4440 msgr2=0x559f17cb6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:07:15.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.608+0000 7fabc0ff9700 20 reqs_thread_entry: start 2019-12-11T17:07:15.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.608+0000 7fabba7fc700 20 reqs_thread_entry: start 2019-12-11T17:07:15.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.608+0000 7fac8db1eec0 20 init_complete bucket index max shards: 0 2019-12-11T17:07:15.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.608+0000 7fabb8ff9700 20 reqs_thread_entry: start 2019-12-11T17:07:15.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.608+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:07:15.684 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.681+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:15.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.704+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:15.723 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.720+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.788+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:07:15.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.843+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:15.863 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.861+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:15.873 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.870+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:15.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:15.919+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:07:16.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.000+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.024+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:16.038 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.036+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.098 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.095+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:07:16.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.166+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.218 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.215+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:16.235 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.233+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.275+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:07:16.332 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.329+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.345+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.352+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.397 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.394+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:07:16.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.481+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.517 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.514+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:16.536 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.532+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.580 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.578+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:07:16.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.627+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.644+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:16.658 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.655+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.688+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:07:16.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.739+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.765 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.762+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:16.779 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.775+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.816 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.813+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:07:16.880 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.876+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.896+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.908+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:16.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:16.946+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:07:17.010 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.006+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.031+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:17.044 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.040+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.102 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.099+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:07:17.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.170+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.192+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:17.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.201+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.256 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.252+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:07:17.325 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.321+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.348 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.344+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:17.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.354+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.417 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.413+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:07:17.474 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.470+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.496+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:17.515 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.512+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.540+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:07:17.583 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.579+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.599 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.594+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:17.606 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.602+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.635+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:07:17.678 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.674+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.693 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.689+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:17.701 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.698+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.732 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.728+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:07:17.818 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.814+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.846 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.842+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.855+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:17.909 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:17.905+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:07:18.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.003+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.030+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:18.050 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.046+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.122 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.117+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:07:18.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.180+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.218 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.214+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:18.237 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.234+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.285 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.281+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:07:18.445 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.441+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.463 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.460+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:18.477 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.473+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.516 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.513+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:07:18.559 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.555+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.573 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.569+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:18.581 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.578+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.614+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:07:18.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.668+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.686 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.683+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:18.695 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.691+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.733+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:07:18.792 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.788+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.808+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:18.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.820+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.871 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.867+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:07:18.934 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.930+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:18.953 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.950+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:18.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:18.959+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.003+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:07:19.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.049+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.065+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:19.078 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.074+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.114+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:07:19.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.156+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.172+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:19.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.181+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.213+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:07:19.299 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.294+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.319 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.315+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:19.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.325+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.378 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.374+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:07:19.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.471+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.501 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.498+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:19.518 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.514+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.569 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.566+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:07:19.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.635+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.661+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:19.678 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.674+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.717 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.713+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:07:19.807 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.803+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.841 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.837+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:19.859 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.855+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:19.937 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.933+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:07:20.000 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:19.996+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:20.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.019+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:20.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.031+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:20.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.070+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:07:20.121 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.117+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:20.136 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.132+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:20.145 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.141+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:20.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.179+0000 7fac8db1eec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:07:20.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.285+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:20.313 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.309+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:20.328 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.324+0000 7fac8db1eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:20.439 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.435+0000 7fac8db1eec0 20 remove_watcher() i=0 2019-12-11T17:07:20.439 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.435+0000 7fac8db1eec0 2 removed watcher, disabling cache 2019-12-11T17:07:20.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.483+0000 7fac8db1eec0 20 remove_watcher() i=1 2019-12-11T17:07:20.530 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.526+0000 7fac8db1eec0 20 remove_watcher() i=2 2019-12-11T17:07:20.583 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.579+0000 7fac8db1eec0 20 remove_watcher() i=3 2019-12-11T17:07:20.619 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.615+0000 7fac8db1eec0 20 remove_watcher() i=4 2019-12-11T17:07:20.660 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.656+0000 7fac8db1eec0 20 remove_watcher() i=5 2019-12-11T17:07:20.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.697+0000 7fac8db1eec0 20 remove_watcher() i=6 2019-12-11T17:07:20.742 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.738+0000 7fac8db1eec0 20 remove_watcher() i=7 2019-12-11T17:07:20.749 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:07:20.863 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.859+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:20.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.881+0000 7f1da8bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:20.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.881+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:20.892 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.889+0000 7f1da8bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:20.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.889+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:20.896 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.892+0000 7f1da8bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:20.896 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.892+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:20.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:20.901+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:21.078 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.075+0000 7f1da8bc7ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:07:21.079 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.075+0000 7f1da8bc7ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:07:21.079 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.075+0000 7f1da8bc7ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:07:21.079 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.075+0000 7f1da8bc7ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:07:21.079 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.075+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.090 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.086+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:21.090 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.086+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.103+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:07:21.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.103+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.115+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:21.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.115+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.132 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.129+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:07:21.133 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.129+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.141 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.138+0000 7f1da8bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:21.142 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.138+0000 7f1da8bc7ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:07:21.142 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.138+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.150 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.146+0000 7f1da8bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:21.150 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.146+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.155+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:21.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.155+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.172+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:07:21.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.172+0000 7f1da8bc7ec0 20 zonegroup default 2019-12-11T17:07:21.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.172+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.180+0000 7f1da8bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:21.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.180+0000 7f1da8bc7ec0 10 Cannot find current period zone using local zone 2019-12-11T17:07:21.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.180+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.185+0000 7f1da8bc7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:21.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.185+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.194+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:21.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.194+0000 7f1da8bc7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:21.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.213+0000 7f1da8bc7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:07:21.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.213+0000 7f1da8bc7ec0 20 zone default found 2019-12-11T17:07:21.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.213+0000 7f1da8bc7ec0 20 started sync module instance, tier type = 2019-12-11T17:07:21.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.213+0000 7f1da8bc7ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:07:21.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.644+0000 7f1da8bc7ec0 20 add_watcher() i=0 2019-12-11T17:07:21.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.644+0000 7f1da8bc7ec0 20 add_watcher() i=1 2019-12-11T17:07:21.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.644+0000 7f1da8bc7ec0 20 add_watcher() i=2 2019-12-11T17:07:21.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.644+0000 7f1da8bc7ec0 20 add_watcher() i=3 2019-12-11T17:07:21.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.644+0000 7f1da8bc7ec0 20 add_watcher() i=4 2019-12-11T17:07:21.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.644+0000 7f1da8bc7ec0 20 add_watcher() i=5 2019-12-11T17:07:21.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.644+0000 7f1da8bc7ec0 20 add_watcher() i=6 2019-12-11T17:07:21.693 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.683+0000 7f1da8bc7ec0 20 add_watcher() i=7 2019-12-11T17:07:21.693 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.683+0000 7f1da8bc7ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:07:21.693 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.683+0000 7f1d667dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:07:21.694 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.690+0000 7f1d95df1700 0 -- 172.21.15.79:0/3591601617 >> v1:172.21.15.79:6789/0 conn(0x564957a55d00 legacy=0x564957a46170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:07:21.740 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:21.736+0000 7f1d965f2700 0 -- 172.21.15.79:0/3742358266 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564957a6c4a0 msgr2=0x564957a46ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:07:23.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.115+0000 7f1cdcff9700 20 reqs_thread_entry: start 2019-12-11T17:07:23.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.116+0000 7f1cd67fc700 20 reqs_thread_entry: start 2019-12-11T17:07:23.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.116+0000 7f1da8bc7ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:07:23.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.116+0000 7f1cd4ff9700 20 reqs_thread_entry: start 2019-12-11T17:07:23.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.116+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:07:23.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.198+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.237 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.233+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:23.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.249+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.314 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.311+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:07:23.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.390+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.423 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.420+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:23.445 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.443+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.481 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.478+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:07:23.519 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.516+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.536 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.533+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:23.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.546+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.589 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.586+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:07:23.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.660+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.693 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.690+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:23.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.710+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.758+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:07:23.815 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.812+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.836 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.834+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:23.850 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.847+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.890 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.887+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:07:23.937 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.934+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:23.959 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.956+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:23.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:23.968+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.008 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.005+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:07:24.059 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.056+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.095 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.092+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.107+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.146 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.144+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:07:24.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.194+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.209+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.220 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.217+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.250 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.247+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:07:24.295 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.293+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.309+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.317+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.352+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:07:24.402 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.399+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.418+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.427 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.425+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.456+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:07:24.507 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.504+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.530+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.542 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.539+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.575+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:07:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.611+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.632 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.630+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.641 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.638+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.670+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:07:24.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.724+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.740+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.751 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.749+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.795 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.792+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:07:24.837 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.835+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.849+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.859 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.857+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.891+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:07:24.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.932+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:24.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.948+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:24.970 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:24.967+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.000+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:07:25.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.044+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.062 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.060+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.068+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.103 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.100+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:07:25.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.164+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.186+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.200+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.244 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.241+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:07:25.296 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.293+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.315+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.328 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.325+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.365 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.362+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:07:25.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.405+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.424 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.421+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.433 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.431+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.471 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.469+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:07:25.514 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.511+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.526+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.536 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.534+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.559+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:07:25.601 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.598+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.613+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.621+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.654 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.651+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:07:25.714 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.711+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.728+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.737 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.735+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.792 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.790+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:07:25.844 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.841+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.860 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.858+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.875 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.871+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.919+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:07:25.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.969+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:25.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.985+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:25.996 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:25.993+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.044 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.041+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:07:26.086 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.083+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.097+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:26.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.105+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.142 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.139+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:07:26.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.189+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.203+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:26.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.212+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.243+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:07:26.341 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.339+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.360 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.357+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:26.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.366+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.413+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:07:26.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.482+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.542 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.539+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:26.559 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.557+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.597 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.594+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:07:26.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.636+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.650+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:26.661 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.658+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.691 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.689+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:07:26.769 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.766+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.799 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.796+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.814+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:26.866 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.863+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:07:26.995 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:26.992+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:27.009 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.007+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:27.019 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.017+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:27.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.050+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:07:27.093 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.090+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:27.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.105+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:27.116 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.114+0000 7f1da8bc7ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:27.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.195+0000 7f1da8bc7ec0 20 remove_watcher() i=0 2019-12-11T17:07:27.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.195+0000 7f1da8bc7ec0 2 removed watcher, disabling cache 2019-12-11T17:07:27.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.245+0000 7f1da8bc7ec0 20 remove_watcher() i=1 2019-12-11T17:07:27.281 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.279+0000 7f1da8bc7ec0 20 remove_watcher() i=2 2019-12-11T17:07:27.358 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.355+0000 7f1da8bc7ec0 20 remove_watcher() i=3 2019-12-11T17:07:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.398+0000 7f1da8bc7ec0 20 remove_watcher() i=4 2019-12-11T17:07:27.440 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.437+0000 7f1da8bc7ec0 20 remove_watcher() i=5 2019-12-11T17:07:27.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.486+0000 7f1da8bc7ec0 20 remove_watcher() i=6 2019-12-11T17:07:27.545 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.542+0000 7f1da8bc7ec0 20 remove_watcher() i=7 2019-12-11T17:07:27.551 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:07:27.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.663+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.693+0000 7f7fa3931ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:27.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.693+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.703 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.701+0000 7f7fa3931ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:27.704 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.701+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.705+0000 7f7fa3931ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:27.708 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.705+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.714 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.711+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:27.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.910+0000 7f7fa3931ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:07:27.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.910+0000 7f7fa3931ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:07:27.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.910+0000 7f7fa3931ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:07:27.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.910+0000 7f7fa3931ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:07:27.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.910+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.930 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.927+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:27.930 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.927+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.948 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.945+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:07:27.948 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.945+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.960 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.958+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:27.960 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.958+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.969+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:07:27.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.969+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.980 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.977+0000 7f7fa3931ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:27.980 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.977+0000 7f7fa3931ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:07:27.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.977+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.984+0000 7f7fa3931ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:27.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.984+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:27.997 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.995+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:27.998 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:27.995+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:28.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.012+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:07:28.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.012+0000 7f7fa3931ec0 20 zonegroup default 2019-12-11T17:07:28.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.012+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.017+0000 7f7fa3931ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.017+0000 7f7fa3931ec0 10 Cannot find current period zone using local zone 2019-12-11T17:07:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.017+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:28.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.023+0000 7f7fa3931ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:07:28.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.023+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:28.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.032+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:07:28.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.032+0000 7f7fa3931ec0 20 rados->read ofs=0 len=0 2019-12-11T17:07:28.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.043+0000 7f7fa3931ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:07:28.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.043+0000 7f7fa3931ec0 20 zone default found 2019-12-11T17:07:28.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.043+0000 7f7fa3931ec0 20 started sync module instance, tier type = 2019-12-11T17:07:28.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.043+0000 7f7fa3931ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:07:28.473 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.470+0000 7f7fa3931ec0 20 add_watcher() i=0 2019-12-11T17:07:28.473 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.470+0000 7f7fa3931ec0 20 add_watcher() i=1 2019-12-11T17:07:28.473 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.470+0000 7f7fa3931ec0 20 add_watcher() i=2 2019-12-11T17:07:28.473 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.470+0000 7f7fa3931ec0 20 add_watcher() i=3 2019-12-11T17:07:28.473 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.470+0000 7f7fa3931ec0 20 add_watcher() i=4 2019-12-11T17:07:28.473 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.470+0000 7f7fa3931ec0 20 add_watcher() i=5 2019-12-11T17:07:28.502 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.499+0000 7f7fa3931ec0 20 add_watcher() i=6 2019-12-11T17:07:28.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.506+0000 7f7fa3931ec0 20 add_watcher() i=7 2019-12-11T17:07:28.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.506+0000 7f7fa3931ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:07:28.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.507+0000 7f7f617e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:07:28.524 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.521+0000 7f7f90b5b700 0 -- 172.21.15.79:0/3742065256 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x5599493dc800 msgr2=0x5599493b98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:07:28.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:28.554+0000 7f7f9035a700 0 -- 172.21.15.79:0/1425227415 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x5599493f00e0 msgr2=0x5599493f2500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:07:30.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.347+0000 7f7ed37fe700 20 reqs_thread_entry: start 2019-12-11T17:07:30.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.347+0000 7f7ed17fa700 20 reqs_thread_entry: start 2019-12-11T17:07:30.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.347+0000 7f7fa3931ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:07:30.352 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.348+0000 7f7ebf7fe700 20 reqs_thread_entry: start 2019-12-11T17:07:30.352 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.348+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:07:30.404 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.401+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.419 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.416+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:30.431 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.424+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.466 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.463+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:07:30.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.523+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.547+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:30.560 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.557+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.619 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.616+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:07:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.675+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.697+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:30.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.708+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.739 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.737+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:07:30.813 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.810+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.839+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:30.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.850+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:30.925 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.921+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:07:30.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:30.973+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.000+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:31.019 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.017+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.059+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:07:31.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.099+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.116 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.113+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:31.125 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.122+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.154+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:07:31.239 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.236+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.265+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:31.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.281+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.338+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:07:31.421 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.418+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.451 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.448+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:31.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.464+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.521+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:07:31.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.606+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.625 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.623+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:31.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.633+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.673+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:07:31.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.743+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.771+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:31.784 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.782+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.830 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.827+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:07:31.879 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.872+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.891 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.888+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:31.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.896+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:31.933 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.930+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:07:32.000 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:31.997+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.021 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.012+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:32.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.021+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.064+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:07:32.147 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.144+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.180+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:32.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.199+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.270 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.267+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:07:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.343+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.371+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:32.386 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.384+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.440+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:07:32.514 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.511+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.540+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:32.564 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.561+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.618 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.615+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:07:32.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.711+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.729 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.726+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:32.737 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.735+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.765+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:07:32.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.822+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.853 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.851+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.865+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.914+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:07:32.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.962+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.980+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:32.991+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.038 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.035+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:07:33.082 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.080+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.098 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.095+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.105+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.144 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.141+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:07:33.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.179+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.195+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.203+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.235+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:07:33.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.275+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.295 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.292+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.300+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.332+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:07:33.371 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.369+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.388 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.385+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.393+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.423+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:07:33.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.508+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.539 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.536+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.553 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.551+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.610 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.607+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:07:33.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.667+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.685+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.700 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.697+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.733 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.730+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:07:33.781 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.778+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.799 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.796+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.808 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.805+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.847 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.844+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:07:33.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.898+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.918 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.915+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:33.928 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.925+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:33.960 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:33.957+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:07:34.085 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.083+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.098+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:34.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.107+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.137 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.134+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:07:34.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.194+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.209+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:34.220 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.217+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.260+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:07:34.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.327+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.352 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.349+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.364+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.416 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.413+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:07:34.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.473+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.497 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.495+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:34.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.505+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.554 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.551+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:07:34.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.601+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.621 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.618+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:34.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.626+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.667+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:07:34.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.710+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.729 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.727+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:07:34.738 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.735+0000 7f7fa3931ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:07:34.838 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.835+0000 7f7fa3931ec0 20 remove_watcher() i=0 2019-12-11T17:07:34.839 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.835+0000 7f7fa3931ec0 2 removed watcher, disabling cache 2019-12-11T17:07:34.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.891+0000 7f7fa3931ec0 20 remove_watcher() i=1 2019-12-11T17:07:34.942 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:34.939+0000 7f7fa3931ec0 20 remove_watcher() i=2 2019-12-11T17:07:35.005 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:35.002+0000 7f7fa3931ec0 20 remove_watcher() i=3 2019-12-11T17:07:35.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:35.060+0000 7f7fa3931ec0 20 remove_watcher() i=4 2019-12-11T17:07:35.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:35.114+0000 7f7fa3931ec0 20 remove_watcher() i=5 2019-12-11T17:07:35.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:35.170+0000 7f7fa3931ec0 20 remove_watcher() i=6 2019-12-11T17:07:35.233 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:07:35.230+0000 7f7fa3931ec0 20 remove_watcher() i=7 2019-12-11T17:07:35.241 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests BucketTest 2019-12-11T17:07:36.048 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :clean 2019-12-11T17:07:36.545 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :compileJava 2019-12-11T17:07:36.545 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :processResources 2019-12-11T17:07:36.546 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :classes 2019-12-11T17:07:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:36.846 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :compileTestJava 2019-12-11T17:07:36.847 INFO:teuthology.orchestra.run.smithi079.stdout:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2019-12-11T17:07:36.847 INFO:teuthology.orchestra.run.smithi079.stdout:Note: Recompile with -Xlint:deprecation for details. 2019-12-11T17:07:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:36.847 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :processTestResources NO-SOURCE 2019-12-11T17:07:36.848 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :testClasses 2019-12-11T17:07:39.345 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:39.346 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :test 2019-12-11T17:07:39.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:39.346 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty STARTED 2019-12-11T17:07:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:41.445 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty PASSED 2019-12-11T17:07:45.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:45.845 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch STARTED 2019-12-11T17:07:47.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:47.845 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch PASSED 2019-12-11T17:07:53.345 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:53.345 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable STARTED 2019-12-11T17:07:55.245 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:55.245 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable PASSED 2019-12-11T17:07:59.345 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:07:59.345 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete STARTED 2019-12-11T17:08:05.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:05.845 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete PASSED 2019-12-11T17:08:07.345 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:07.345 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty STARTED 2019-12-11T17:08:11.745 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:11.745 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty PASSED 2019-12-11T17:08:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:17.145 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist STARTED 2019-12-11T17:08:17.345 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:17.345 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist PASSED 2019-12-11T17:08:18.645 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:18.645 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct STARTED 2019-12-11T17:08:24.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:24.845 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct PASSED 2019-12-11T17:08:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:36.545 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty STARTED 2019-12-11T17:08:39.345 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:39.345 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty PASSED 2019-12-11T17:08:44.345 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:44.345 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketNotExist STARTED 2019-12-11T17:08:44.445 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:44.445 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Bucket tests > BucketTest.testBucketNotExist PASSED 2019-12-11T17:08:44.891 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:44.891 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------- 2019-12-11T17:08:44.892 INFO:teuthology.orchestra.run.smithi079.stdout:| Results: SUCCESS (9 tests, 9 successes, 0 failures, 0 skipped) | 2019-12-11T17:08:44.892 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------- 2019-12-11T17:08:44.892 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:08:44.892 INFO:teuthology.orchestra.run.smithi079.stdout:BUILD SUCCESSFUL in 1m 9s 2019-12-11T17:08:44.893 INFO:teuthology.orchestra.run.smithi079.stdout:5 actionable tasks: 5 executed 2019-12-11T17:08:44.898 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:08:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.027+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.070 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.066+0000 7fde9bdcfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:45.070 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.066+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.078 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.075+0000 7fde9bdcfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:45.078 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.075+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.083 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.079+0000 7fde9bdcfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:45.083 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.079+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.094 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.090+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.248+0000 7fde9bdcfec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:08:45.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.248+0000 7fde9bdcfec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:08:45.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.248+0000 7fde9bdcfec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:08:45.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.248+0000 7fde9bdcfec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:08:45.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.248+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.257+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.257+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.268+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:08:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.268+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.280 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.277+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.277+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.298+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:08:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.298+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.303+0000 7fde9bdcfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.303+0000 7fde9bdcfec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:08:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.303+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.311 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.308+0000 7fde9bdcfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.308+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.317+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.318+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.326+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:08:45.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.326+0000 7fde9bdcfec0 20 zonegroup default 2019-12-11T17:08:45.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.326+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.333+0000 7fde9bdcfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:45.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.333+0000 7fde9bdcfec0 10 Cannot find current period zone using local zone 2019-12-11T17:08:45.337 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.333+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.340+0000 7fde9bdcfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:45.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.340+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.354 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.350+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:45.354 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.350+0000 7fde9bdcfec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:45.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.364+0000 7fde9bdcfec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:08:45.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.364+0000 7fde9bdcfec0 20 zone default found 2019-12-11T17:08:45.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.364+0000 7fde9bdcfec0 20 started sync module instance, tier type = 2019-12-11T17:08:45.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.364+0000 7fde9bdcfec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:08:45.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.702+0000 7fde9bdcfec0 20 add_watcher() i=0 2019-12-11T17:08:45.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.702+0000 7fde9bdcfec0 20 add_watcher() i=1 2019-12-11T17:08:45.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.702+0000 7fde9bdcfec0 20 add_watcher() i=2 2019-12-11T17:08:45.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.702+0000 7fde9bdcfec0 20 add_watcher() i=3 2019-12-11T17:08:45.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.702+0000 7fde9bdcfec0 20 add_watcher() i=4 2019-12-11T17:08:45.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.702+0000 7fde9bdcfec0 20 add_watcher() i=5 2019-12-11T17:08:45.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.721+0000 7fde9bdcfec0 20 add_watcher() i=6 2019-12-11T17:08:45.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.750+0000 7fde9bdcfec0 20 add_watcher() i=7 2019-12-11T17:08:45.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.750+0000 7fde9bdcfec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:08:45.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.750+0000 7fde58fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:08:45.769 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.766+0000 7fde88ff9700 0 -- 172.21.15.79:0/638725271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5624cec03010 msgr2=0x5624cebec940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:08:45.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:45.805+0000 7fde897fa700 0 -- 172.21.15.79:0/1357234785 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x5624cebd8f90 msgr2=0x5624cebd8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:08:47.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.259+0000 7fddcaffd700 20 reqs_thread_entry: start 2019-12-11T17:08:47.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.259+0000 7fddc8ff9700 20 reqs_thread_entry: start 2019-12-11T17:08:47.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.259+0000 7fde9bdcfec0 20 init_complete bucket index max shards: 0 2019-12-11T17:08:47.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.259+0000 7fddbeffd700 20 reqs_thread_entry: start 2019-12-11T17:08:47.264 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.259+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:08:47.306 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.302+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.318+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:47.330 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.326+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.358+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:08:47.403 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.399+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.416+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:47.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.425+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.458+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:08:47.539 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.535+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.564 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.560+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:47.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.573+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.627 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.623+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:08:47.691 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.687+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.709 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.705+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:47.718 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.714+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.757 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.753+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:08:47.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.824+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.854 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.850+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:47.861 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.858+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:47.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:47.919+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:08:48.017 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.013+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.043+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:48.067 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.063+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.107+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:08:48.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.166+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.189+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:48.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.201+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.243+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:08:48.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.299+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.318 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.314+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:48.326 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.322+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.379 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.375+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:08:48.473 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.462+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.507 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.503+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:48.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.525+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.584 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.580+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:08:48.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.641+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.665 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.661+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:48.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.672+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.723 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.720+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:08:48.830 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.826+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.848+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:48.860 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.857+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.910+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:08:48.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.968+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.986+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.000 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:48.996+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.031+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:08:49.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.115+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.156+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.172+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.242 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.238+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:08:49.311 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.307+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.332 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.328+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.344 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.341+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.390 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.387+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:08:49.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.432+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.445+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.458 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.454+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.499 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.495+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:08:49.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.540+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.559+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.567+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.605 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.601+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:08:49.646 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.643+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.661 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.657+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.666+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.703 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.699+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:08:49.744 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.740+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.754+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.763+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.800 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.796+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:08:49.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.878+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:49.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.917+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:49.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:49.944+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.005 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.001+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:08:50.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.069+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.092+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.105+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.150+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:08:50.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.201+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.223 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.219+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:50.235 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.231+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.273 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.269+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:08:50.337 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.334+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.351+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:50.365 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.362+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.392+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:08:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.458+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.487+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:50.499 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.495+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.527+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:08:50.595 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.591+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.610+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:50.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.621+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.671 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.667+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:08:50.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.727+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.755 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.751+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:50.764 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.760+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.816 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.812+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:08:50.879 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.876+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.895 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.891+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:50.902 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.899+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:50.948 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:50.944+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:08:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.012+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.036 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.032+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:51.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.043+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.087 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.083+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:08:51.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.126+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.145 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.141+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:51.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.150+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.197+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:08:51.255 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.251+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.265+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:51.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.275+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.323 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.319+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:08:51.411 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.407+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.439+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:51.460 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.458+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.519+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:08:51.592 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.590+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.612+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:51.625 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.623+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.668 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.666+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:08:51.783 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.780+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.824+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:51.839 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.836+0000 7fde9bdcfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:51.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.947+0000 7fde9bdcfec0 20 remove_watcher() i=0 2019-12-11T17:08:51.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:51.947+0000 7fde9bdcfec0 2 removed watcher, disabling cache 2019-12-11T17:08:52.032 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.029+0000 7fde9bdcfec0 20 remove_watcher() i=1 2019-12-11T17:08:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.070+0000 7fde9bdcfec0 20 remove_watcher() i=2 2019-12-11T17:08:52.148 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.145+0000 7fde9bdcfec0 20 remove_watcher() i=3 2019-12-11T17:08:52.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.208+0000 7fde9bdcfec0 20 remove_watcher() i=4 2019-12-11T17:08:52.270 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.267+0000 7fde9bdcfec0 20 remove_watcher() i=5 2019-12-11T17:08:52.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.331+0000 7fde9bdcfec0 20 remove_watcher() i=6 2019-12-11T17:08:52.399 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.396+0000 7fde9bdcfec0 20 remove_watcher() i=7 2019-12-11T17:08:52.412 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:08:52.609 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.606+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.638 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.635+0000 7f62e1c26ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:52.639 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.635+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.645+0000 7f62e1c26ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.645+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.652 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.650+0000 7f62e1c26ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:52.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.650+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.660 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.658+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:52.819 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.816+0000 7f62e1c26ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:08:52.819 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.816+0000 7f62e1c26ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:08:52.819 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.816+0000 7f62e1c26ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:08:52.819 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.816+0000 7f62e1c26ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:08:52.819 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.816+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.830 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.827+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:52.830 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.827+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.848 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.845+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:08:52.848 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.845+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.853+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.853+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.876 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.870+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:08:52.876 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.870+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.875+0000 7f62e1c26ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:52.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.875+0000 7f62e1c26ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:08:52.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.875+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.882+0000 7f62e1c26ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:52.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.882+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.895 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.893+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:52.896 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.893+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.913+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:08:52.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.913+0000 7f62e1c26ec0 20 zonegroup default 2019-12-11T17:08:52.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.913+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.922 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.920+0000 7f62e1c26ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:52.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.920+0000 7f62e1c26ec0 10 Cannot find current period zone using local zone 2019-12-11T17:08:52.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.920+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.928 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.925+0000 7f62e1c26ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:08:52.928 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.925+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.938 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.935+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:08:52.938 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.935+0000 7f62e1c26ec0 20 rados->read ofs=0 len=0 2019-12-11T17:08:52.953 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.950+0000 7f62e1c26ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:08:52.953 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.950+0000 7f62e1c26ec0 20 zone default found 2019-12-11T17:08:52.953 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.950+0000 7f62e1c26ec0 20 started sync module instance, tier type = 2019-12-11T17:08:52.953 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:52.950+0000 7f62e1c26ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:08:53.319 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.317+0000 7f62e1c26ec0 20 add_watcher() i=0 2019-12-11T17:08:53.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.317+0000 7f62e1c26ec0 20 add_watcher() i=1 2019-12-11T17:08:53.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.317+0000 7f62e1c26ec0 20 add_watcher() i=2 2019-12-11T17:08:53.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.317+0000 7f62e1c26ec0 20 add_watcher() i=3 2019-12-11T17:08:53.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.317+0000 7f62e1c26ec0 20 add_watcher() i=4 2019-12-11T17:08:53.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.317+0000 7f62e1c26ec0 20 add_watcher() i=5 2019-12-11T17:08:53.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.317+0000 7f62e1c26ec0 20 add_watcher() i=6 2019-12-11T17:08:53.352 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.350+0000 7f62e1c26ec0 20 add_watcher() i=7 2019-12-11T17:08:53.353 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.350+0000 7f62e1c26ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:08:53.353 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.350+0000 7f629f7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:08:53.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.353+0000 7f62cee50700 0 -- >> v1:172.21.15.199:6789/0 conn(0x561c0b9d6090 legacy=0x561c0b9d4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:08:53.406 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:53.403+0000 7f62cf651700 0 -- 172.21.15.79:0/627384657 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7f6210022480 msgr2=0x7f6210024990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:08:54.800 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.797+0000 7f620dffb700 20 reqs_thread_entry: start 2019-12-11T17:08:54.800 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.797+0000 7f61ff7fe700 20 reqs_thread_entry: start 2019-12-11T17:08:54.800 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.797+0000 7f62e1c26ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:08:54.801 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.798+0000 7f61fdffb700 20 reqs_thread_entry: start 2019-12-11T17:08:54.801 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.798+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:08:54.880 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.877+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:54.904 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.901+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:54.916 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.913+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:54.969 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:54.967+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:08:55.049 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.047+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.087 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.084+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:55.102 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.100+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.180+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:08:55.245 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.242+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.265+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:55.279 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.277+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.312+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:08:55.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.393+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.433 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.430+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:55.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.445+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.499 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.497+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:08:55.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.541+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.561 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.558+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:55.570 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.567+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.611 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.609+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:08:55.651 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.648+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.664+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:55.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.672+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.701+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:08:55.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.782+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.812 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.809+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:55.828 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.825+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:55.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.896+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:08:55.983 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:55.980+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.012 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.009+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:56.032 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.030+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.087 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.085+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:08:56.140 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.137+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.153+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:56.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.162+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.215 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.212+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:08:56.295 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.292+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.318+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:56.337 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.334+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.386+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:08:56.447 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.444+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.478 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.475+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:56.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.488+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.539 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.536+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:08:56.602 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.599+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.625 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.622+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:56.649 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.647+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.733 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.730+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:08:56.798 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.795+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.821 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.818+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:56.835 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.832+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.881 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.878+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:08:56.955 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.952+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.978+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:56.994 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:56.991+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.042 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.039+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:08:57.116 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.113+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.135 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.133+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:57.148 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.145+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.205+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:08:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.265+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.288 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.285+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:57.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.297+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.340+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:08:57.417 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.414+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.445 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.443+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:57.464 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.461+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.530 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.527+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:08:57.587 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.584+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.628 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.619+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:57.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.640+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.686+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:08:57.743 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.740+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.762 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.759+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:57.770 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.767+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.819 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.816+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:08:57.876 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.873+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.911+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.921+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:57.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:57.963+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:08:58.006 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.004+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.020+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.031 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.028+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.060+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:08:58.105 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.103+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.121+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.129+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.167+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:08:58.256 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.253+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.286+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.302 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.299+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.388 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.385+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:08:58.447 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.444+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.472+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.480+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.542+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:08:58.614 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.611+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.644 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.641+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.654 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.651+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.696+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:08:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.742+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.758+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.769 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.766+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.800+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:08:58.841 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.839+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.856 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.853+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.864 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.862+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.898 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.895+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:08:58.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.959+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.976+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:58.988 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:58.985+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.033 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.031+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:08:59.096 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.093+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.115 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.112+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:59.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.125+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.169+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:08:59.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.235+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.259 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.256+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.267+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.318 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.315+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:08:59.406 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.403+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.462 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.460+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:59.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.484+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.521 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.519+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:08:59.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.561+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.580 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.578+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:08:59.591 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.588+0000 7f62e1c26ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:08:59.681 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.679+0000 7f62e1c26ec0 20 remove_watcher() i=0 2019-12-11T17:08:59.682 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.679+0000 7f62e1c26ec0 2 removed watcher, disabling cache 2019-12-11T17:08:59.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.728+0000 7f62e1c26ec0 20 remove_watcher() i=1 2019-12-11T17:08:59.771 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.768+0000 7f62e1c26ec0 20 remove_watcher() i=2 2019-12-11T17:08:59.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.822+0000 7f62e1c26ec0 20 remove_watcher() i=3 2019-12-11T17:08:59.891 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.888+0000 7f62e1c26ec0 20 remove_watcher() i=4 2019-12-11T17:08:59.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.917+0000 7f62e1c26ec0 20 remove_watcher() i=5 2019-12-11T17:08:59.953 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.951+0000 7f62e1c26ec0 20 remove_watcher() i=6 2019-12-11T17:08:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:08:59.988+0000 7f62e1c26ec0 20 remove_watcher() i=7 2019-12-11T17:09:00.001 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:09:00.132 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.126+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.162+0000 7f19ffbceec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:00.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.162+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.172+0000 7f19ffbceec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:00.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.172+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.177+0000 7f19ffbceec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:00.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.177+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.186+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:00.350 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.347+0000 7f19ffbceec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:09:00.350 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.347+0000 7f19ffbceec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:09:00.350 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.347+0000 7f19ffbceec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:09:00.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.347+0000 7f19ffbceec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:09:00.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.347+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.356+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:00.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.356+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.379 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.376+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:09:00.379 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.376+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.386+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:00.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.386+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.406 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.403+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:09:00.406 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.403+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.414 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.411+0000 7f19ffbceec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:00.414 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.411+0000 7f19ffbceec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:09:00.414 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.411+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.419 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.417+0000 7f19ffbceec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:00.419 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.417+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.427 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.424+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:00.427 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.424+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.441 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.439+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:09:00.441 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.439+0000 7f19ffbceec0 20 zonegroup default 2019-12-11T17:09:00.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.439+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.446 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.444+0000 7f19ffbceec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:00.446 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.444+0000 7f19ffbceec0 10 Cannot find current period zone using local zone 2019-12-11T17:09:00.447 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.444+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.451 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.448+0000 7f19ffbceec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:00.451 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.448+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.456+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.456+0000 7f19ffbceec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:00.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.473+0000 7f19ffbceec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:09:00.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.473+0000 7f19ffbceec0 20 zone default found 2019-12-11T17:09:00.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.473+0000 7f19ffbceec0 20 started sync module instance, tier type = 2019-12-11T17:09:00.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.473+0000 7f19ffbceec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:09:00.826 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.823+0000 7f19ffbceec0 20 add_watcher() i=0 2019-12-11T17:09:00.826 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.823+0000 7f19ffbceec0 20 add_watcher() i=1 2019-12-11T17:09:00.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.823+0000 7f19ffbceec0 20 add_watcher() i=2 2019-12-11T17:09:00.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.823+0000 7f19ffbceec0 20 add_watcher() i=3 2019-12-11T17:09:00.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.823+0000 7f19ffbceec0 20 add_watcher() i=4 2019-12-11T17:09:00.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.823+0000 7f19ffbceec0 20 add_watcher() i=5 2019-12-11T17:09:00.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.823+0000 7f19ffbceec0 20 add_watcher() i=6 2019-12-11T17:09:00.869 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.866+0000 7f19ffbceec0 20 add_watcher() i=7 2019-12-11T17:09:00.869 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.866+0000 7f19ffbceec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:09:00.869 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.866+0000 7f19bd7e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:09:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.874+0000 7f19ed5f9700 0 -- 172.21.15.79:0/3026543407 >> v1:172.21.15.79:6789/0 conn(0x564c36b000f0 legacy=0x564c36b1dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-11T17:09:00.918 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:00.915+0000 7f19ec5f7700 0 -- 172.21.15.79:0/3971575061 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x564c36b4cc90 msgr2=0x564c36b4f0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:09:02.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.445+0000 7f192f7fe700 20 reqs_thread_entry: start 2019-12-11T17:09:02.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.445+0000 7f192d7fa700 20 reqs_thread_entry: start 2019-12-11T17:09:02.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.445+0000 7f19ffbceec0 20 init_complete bucket index max shards: 0 2019-12-11T17:09:02.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.446+0000 7f191b7fe700 20 reqs_thread_entry: start 2019-12-11T17:09:02.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.446+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:09:02.488 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.485+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.502 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.499+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.508+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.548 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.545+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:09:02.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.593+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.610 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.608+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:02.619 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.616+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.657+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:09:02.713 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.710+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.725+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:02.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.733+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.778 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.775+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:09:02.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.821+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.838 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.835+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:02.847 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.844+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.886 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.883+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:09:02.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.947+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:02.964 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.962+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:02.970+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.027 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.024+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:09:03.075 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.072+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.091 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.088+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.097+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.132 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.130+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:09:03.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.168+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.184+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.195+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.227 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.224+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:09:03.289 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.276+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.308 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.305+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.318 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.315+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.365+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:09:03.430 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.427+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.462 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.459+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.472+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.550 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.547+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:09:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.596+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.618 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.615+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.630 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.627+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.661 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.658+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:09:03.753 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.750+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.783 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.780+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.796 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.793+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.833 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.830+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:09:03.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.879+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.897 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.894+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.903+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.934+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:09:03.971 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.968+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:03.988 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.986+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:03.996 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:03.994+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.024+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:09:04.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.109+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.141 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.138+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:04.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.153+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.209+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:09:04.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.279+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.313 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.310+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:04.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.326+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.386+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:09:04.447 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.444+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.468 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.465+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:04.478 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.475+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.525+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:09:04.588 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.585+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.612 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.610+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:04.629 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.626+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.691 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.688+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:09:04.755 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.753+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.780+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:04.796 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.794+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.834 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.832+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:09:04.889 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.886+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.907+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:04.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.921+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:04.983 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:04.980+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:09:05.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.066+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.099+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:05.117 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.115+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.177+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:09:05.232 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.229+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.257 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.254+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:05.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.267+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.314+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:09:05.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.377+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.417+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:05.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.432+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.469+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:09:05.517 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.514+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.532 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.530+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:05.540 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.538+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.568+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:09:05.651 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.648+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.682+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:05.699 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.697+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.746+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:09:05.795 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.792+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.810 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.807+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:05.818 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.815+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.857 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.855+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:09:05.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.902+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.920 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.917+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:05.928 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.925+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:05.967 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:05.965+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:09:06.010 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.007+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.022+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:06.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.031+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.062+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:09:06.099 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.097+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.114 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.111+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:06.122 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.119+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.153 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.151+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:09:06.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.193+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.211+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:06.225 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.222+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.256 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.253+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:09:06.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.335+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.377+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:06.398 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.395+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.456+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:09:06.555 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.552+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.574+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:06.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.587+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.633+0000 7f19ffbceec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:09:06.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.693+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.719 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.716+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:06.730 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.727+0000 7f19ffbceec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:06.813 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.810+0000 7f19ffbceec0 20 remove_watcher() i=0 2019-12-11T17:09:06.813 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.810+0000 7f19ffbceec0 2 removed watcher, disabling cache 2019-12-11T17:09:06.851 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.848+0000 7f19ffbceec0 20 remove_watcher() i=1 2019-12-11T17:09:06.890 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.888+0000 7f19ffbceec0 20 remove_watcher() i=2 2019-12-11T17:09:06.926 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.924+0000 7f19ffbceec0 20 remove_watcher() i=3 2019-12-11T17:09:06.959 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.956+0000 7f19ffbceec0 20 remove_watcher() i=4 2019-12-11T17:09:06.998 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:06.995+0000 7f19ffbceec0 20 remove_watcher() i=5 2019-12-11T17:09:07.043 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.040+0000 7f19ffbceec0 20 remove_watcher() i=6 2019-12-11T17:09:07.086 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.083+0000 7f19ffbceec0 20 remove_watcher() i=7 2019-12-11T17:09:07.092 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:09:07.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.207+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.232 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.230+0000 7f99dbb6bec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:07.233 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.230+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.240 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.238+0000 7f99dbb6bec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:07.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.238+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.245 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.243+0000 7f99dbb6bec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:07.246 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.243+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.251+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:07.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.639+0000 7f99dbb6bec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:09:07.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.639+0000 7f99dbb6bec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:09:07.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.639+0000 7f99dbb6bec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:09:07.642 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.639+0000 7f99dbb6bec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:09:07.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.639+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.647+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.647+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.666+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:09:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.666+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.674+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:07.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.674+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.694 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.691+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:09:07.694 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.691+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.696+0000 7f99dbb6bec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:07.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.696+0000 7f99dbb6bec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:09:07.699 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.696+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.703 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.701+0000 7f99dbb6bec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:07.704 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.701+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.709+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:07.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.709+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.723+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:09:07.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.723+0000 7f99dbb6bec0 20 zonegroup default 2019-12-11T17:09:07.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.723+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.733 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.730+0000 7f99dbb6bec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:07.733 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.730+0000 7f99dbb6bec0 10 Cannot find current period zone using local zone 2019-12-11T17:09:07.733 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.730+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.737 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.735+0000 7f99dbb6bec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:09:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.735+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.742+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:09:07.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.742+0000 7f99dbb6bec0 20 rados->read ofs=0 len=0 2019-12-11T17:09:07.761 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.758+0000 7f99dbb6bec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:09:07.761 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.758+0000 7f99dbb6bec0 20 zone default found 2019-12-11T17:09:07.762 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.758+0000 7f99dbb6bec0 20 started sync module instance, tier type = 2019-12-11T17:09:07.762 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:07.758+0000 7f99dbb6bec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:09:08.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.115+0000 7f99dbb6bec0 20 add_watcher() i=0 2019-12-11T17:09:08.118 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.115+0000 7f99dbb6bec0 20 add_watcher() i=1 2019-12-11T17:09:08.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.115+0000 7f99dbb6bec0 20 add_watcher() i=2 2019-12-11T17:09:08.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.115+0000 7f99dbb6bec0 20 add_watcher() i=3 2019-12-11T17:09:08.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.115+0000 7f99dbb6bec0 20 add_watcher() i=4 2019-12-11T17:09:08.119 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.115+0000 7f99dbb6bec0 20 add_watcher() i=5 2019-12-11T17:09:08.145 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.143+0000 7f99dbb6bec0 20 add_watcher() i=6 2019-12-11T17:09:08.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.152+0000 7f99dbb6bec0 20 add_watcher() i=7 2019-12-11T17:09:08.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.152+0000 7f99dbb6bec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:09:08.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.152+0000 7f99997e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:09:08.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.155+0000 7f99c9596700 0 -- 172.21.15.79:0/1854096103 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x55afd06195a0 msgr2=0x55afd06387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:09:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:08.196+0000 7f99c8d95700 0 -- 172.21.15.79:0/2096399245 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x55afd0650180 msgr2=0x55afd06525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:09:09.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.732+0000 7f990b7fe700 20 reqs_thread_entry: start 2019-12-11T17:09:09.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.733+0000 7f99097fa700 20 reqs_thread_entry: start 2019-12-11T17:09:09.736 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.733+0000 7f99dbb6bec0 20 init_complete bucket index max shards: 0 2019-12-11T17:09:09.736 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.733+0000 7f98f77fe700 20 reqs_thread_entry: start 2019-12-11T17:09:09.736 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.733+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:09:09.820 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.817+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:09.835 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.833+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:09.844 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.841+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:09.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.896+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:09:09.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.970+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:09.996 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:09.993+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.006 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.002+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.050+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:09:10.114 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.111+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.129+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.141 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.138+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.172+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:09:10.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.210+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.229 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.226+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.236 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.234+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.264+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:09:10.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.312+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.337 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.334+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.354 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.351+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.393+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:09:10.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.473+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.508+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.528+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.599 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.596+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:09:10.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.688+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.718 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.715+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.738+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.783 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.781+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:09:10.831 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.828+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.854 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.851+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.862 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.859+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.900+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:09:10.956 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.954+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:10.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.969+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:10.980 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:10.977+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.019 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.016+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:09:11.070 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.065+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.082 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.080+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:11.091 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.088+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.122+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:09:11.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.195+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.238+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:11.250 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.248+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.302 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.300+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:09:11.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.354+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.369+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:11.381 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.378+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.428 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.425+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:09:11.504 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.501+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.520+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:11.531 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.528+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.586 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.584+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:09:11.624 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.621+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.638 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.636+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:11.646 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.644+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.679 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.677+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:09:11.720 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.717+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.735 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.732+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:11.742 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.740+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.774 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.771+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:09:11.828 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.823+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.838+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:11.849 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.846+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.900 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.897+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:09:11.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.962+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:11.990 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:11.987+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:12.009 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.000+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.048 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.045+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:09:12.129 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.127+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.146 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.143+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:12.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.152+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.193+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:09:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.247+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.266 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.263+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:12.274 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.271+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.312 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.310+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:09:12.367 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.364+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.383+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:12.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.393+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.429 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.427+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:09:12.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.472+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.488+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:12.498 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.496+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.530 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.527+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:09:12.613 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.610+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.650 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.647+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:12.667 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.663+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.737 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.734+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:09:12.796 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.793+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.826 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.823+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:12.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.839+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:12.902 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.899+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:09:12.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.977+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:12.999+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:13.013 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.011+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.084 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.081+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:09:13.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.313+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.349 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.347+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:13.367 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.364+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.425 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.422+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:09:13.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.484+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.508 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.503+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:13.516 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.514+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.546+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:09:13.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.613+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.633+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:13.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.642+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.704+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:09:13.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.761+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.779+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.788+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.828 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.825+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:09:13.873 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.870+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.890+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:13.903 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.901+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:13.949 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:13.947+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:09:14.019 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.016+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:14.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.053+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:14.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.068+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:14.143 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.140+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:09:14.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.193+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:14.216 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.213+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:14.226 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.224+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:14.258 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.256+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:09:14.341 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.339+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:14.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.371+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:09:14.394 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.392+0000 7f99dbb6bec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:09:14.650 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.648+0000 7f99dbb6bec0 20 remove_watcher() i=0 2019-12-11T17:09:14.651 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.648+0000 7f99dbb6bec0 2 removed watcher, disabling cache 2019-12-11T17:09:14.699 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.696+0000 7f99dbb6bec0 20 remove_watcher() i=1 2019-12-11T17:09:14.753 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.750+0000 7f99dbb6bec0 20 remove_watcher() i=2 2019-12-11T17:09:14.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.822+0000 7f99dbb6bec0 20 remove_watcher() i=3 2019-12-11T17:09:14.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.891+0000 7f99dbb6bec0 20 remove_watcher() i=4 2019-12-11T17:09:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:14.954+0000 7f99dbb6bec0 20 remove_watcher() i=5 2019-12-11T17:09:15.022 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:15.019+0000 7f99dbb6bec0 20 remove_watcher() i=6 2019-12-11T17:09:15.073 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:09:15.071+0000 7f99dbb6bec0 20 remove_watcher() i=7 2019-12-11T17:09:15.086 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest 2019-12-11T17:09:15.746 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :clean 2019-12-11T17:09:16.243 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :compileJava 2019-12-11T17:09:16.243 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :processResources 2019-12-11T17:09:16.243 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :classes 2019-12-11T17:09:16.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:16.444 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :compileTestJava 2019-12-11T17:09:16.444 INFO:teuthology.orchestra.run.smithi079.stdout:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2019-12-11T17:09:16.444 INFO:teuthology.orchestra.run.smithi079.stdout:Note: Recompile with -Xlint:deprecation for details. 2019-12-11T17:09:16.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:16.445 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :processTestResources NO-SOURCE 2019-12-11T17:09:16.445 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :testClasses 2019-12-11T17:09:18.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:18.443 INFO:teuthology.orchestra.run.smithi079.stdout:> Task :test 2019-12-11T17:09:18.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:18.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.TestObjectCopyKeyNotFound STARTED 2019-12-11T17:09:23.543 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:23.543 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.TestObjectCopyKeyNotFound PASSED 2019-12-11T17:09:30.838 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:37:17.769 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:30.843 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:37:17.774 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:30.853 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:37:17.784 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:30.859 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:37:17.762 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:30.861 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:37:17.792 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:30.878 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:37:17.781 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:31.019 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:37:17.950 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:31.022 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:37:17.954 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:31.100 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:37:18.003 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:09:31.842 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:31.843 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadExpectNone STARTED 2019-12-11T17:09:34.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:34.843 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadExpectNone PASSED 2019-12-11T17:09:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:41.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadMd5Empty STARTED 2019-12-11T17:09:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:44.143 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadMd5Empty PASSED 2019-12-11T17:09:48.743 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:09:48.743 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadLLAPI STARTED 2019-12-11T17:10:27.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:27.343 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadLLAPI PASSED 2019-12-11T17:10:33.142 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:33.143 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadNotFoundLLAPI STARTED 2019-12-11T17:10:35.543 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:35.543 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadNotFoundLLAPI PASSED 2019-12-11T17:10:39.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:39.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testDownloadHLAPI STARTED 2019-12-11T17:10:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:44.043 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testDownloadHLAPI PASSED 2019-12-11T17:10:52.142 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:52.143 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testDownloadNoBucketHLAPI STARTED 2019-12-11T17:10:52.743 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:52.743 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testDownloadNoBucketHLAPI PASSED 2019-12-11T17:10:53.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:53.343 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testDownloadNoKeyHLAPI STARTED 2019-12-11T17:10:55.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:10:55.043 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testDownloadNoKeyHLAPI PASSED 2019-12-11T17:11:01.142 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:01.143 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1MB STARTED 2019-12-11T17:11:07.643 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:07.643 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1MB PASSED 2019-12-11T17:11:14.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:14.043 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1b STARTED 2019-12-11T17:11:17.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:17.843 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1b PASSED 2019-12-11T17:11:23.943 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:23.943 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1kb STARTED 2019-12-11T17:11:26.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:26.343 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1kb PASSED 2019-12-11T17:11:33.643 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:33.643 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeyNoSSEC STARTED 2019-12-11T17:11:36.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:36.143 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeyNoSSEC PASSED 2019-12-11T17:11:40.942 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:40.943 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECInvalidMd5 STARTED 2019-12-11T17:11:43.943 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:43.943 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECInvalidMd5 PASSED 2019-12-11T17:11:49.642 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:49.643 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoKey STARTED 2019-12-11T17:11:52.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:52.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoKey PASSED 2019-12-11T17:11:58.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:11:58.043 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoMd5 STARTED 2019-12-11T17:12:00.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:12:00.343 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoMd5 PASSED 2019-12-11T17:12:04.242 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:12:04.243 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultiObjectDelete STARTED 2019-12-11T17:12:07.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:12:07.343 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultiObjectDelete PASSED 2019-12-11T17:12:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:12:10.243 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyHLAPIA STARTED 2019-12-11T17:12:20.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:12:20.543 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyHLAPIA PASSED 2019-12-11T17:12:27.643 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:12:27.643 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyMultipleSizesLLAPI STARTED 2019-12-11T17:14:44.061 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:42:30.992 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.072 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:42:31.003 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.079 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:42:31.010 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.085 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:42:31.016 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.091 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:42:30.994 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.098 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:42:31.001 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.251 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:42:31.182 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.307 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:42:31.238 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:44.316 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:42:31.218 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.443 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:42:42.374 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.462 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:42:42.394 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.483 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:42:42.414 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.487 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:42:42.418 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.492 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:42:42.394 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.493 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:42:42.424 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.509 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:42:42.439 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.533 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:42:42.434 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:14:55.563 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:42:42.465 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:15:36.643 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:15:36.643 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyMultipleSizesLLAPI PASSED 2019-12-11T17:15:44.543 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:15:44.543 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCBucketHLAPI STARTED 2019-12-11T17:15:47.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:15:47.143 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCBucketHLAPI PASSED 2019-12-11T17:15:52.943 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:15:52.943 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCKeyHLAPI STARTED 2019-12-11T17:15:56.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:15:56.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCKeyHLAPI PASSED 2019-12-11T17:16:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:16:09.643 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartDownloadHLAPI STARTED 2019-12-11T17:16:20.243 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:16:20.243 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartDownloadHLAPI PASSED 2019-12-11T17:16:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:16:29.343 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartDownloadNoBucketHLAPI STARTED 2019-12-11T17:16:29.743 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:16:29.743 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartDownloadNoBucketHLAPI PASSED 2019-12-11T17:16:30.243 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:16:30.243 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartDownloadWithPauseHLAPI STARTED 2019-12-11T17:16:44.661 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T17:16:44.653+0000 41264700 -1 handle_data failed with exit code -5 2019-12-11T17:16:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:16:45.243 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartDownloadWithPauseHLAPI PASSED 2019-12-11T17:16:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:16:53.543 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadHLAPIAWS4 STARTED 2019-12-11T17:17:19.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:17:19.043 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadHLAPIAWS4 PASSED 2019-12-11T17:17:29.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:17:29.443 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadIncorrectMissingPartLLAPI STARTED 2019-12-11T17:17:49.243 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:17:49.243 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadIncorrectMissingPartLLAPI PASSED 2019-12-11T17:17:55.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:17:55.143 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadMultipleSizesLLAPI STARTED 2019-12-11T17:19:55.240 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:47:42.171 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.244 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:47:42.175 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.251 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:47:42.182 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.254 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:47:42.185 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.263 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:47:42.165 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.276 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:47:42.177 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.418 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:47:42.349 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.439 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:47:42.369 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:19:55.502 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:47:42.403 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:20:43.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:20:43.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadMultipleSizesLLAPI PASSED 2019-12-11T17:20:50.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:20:50.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadOverwriteExistingObjectLLAPI STARTED 2019-12-11T17:21:16.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:21:16.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadOverwriteExistingObjectLLAPI PASSED 2019-12-11T17:21:24.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:21:24.043 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadSmallLLAPI STARTED 2019-12-11T17:21:47.846 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:21:47.846 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadSmallLLAPI PASSED 2019-12-11T17:21:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:21:54.044 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadWithPause STARTED 2019-12-11T17:22:01.243 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:01.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testMultipartUploadWithPause PASSED 2019-12-11T17:22:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:12.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCopyBucketNotFound STARTED 2019-12-11T17:22:13.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:13.044 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCopyBucketNotFound PASSED 2019-12-11T17:22:14.145 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:14.145 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadContentlengthNegative STARTED 2019-12-11T17:22:15.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:15.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadContentlengthNegative PASSED 2019-12-11T17:22:18.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:18.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch STARTED 2019-12-11T17:22:19.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:19.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch PASSED 2019-12-11T17:22:23.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:23.744 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5InvalidShort STARTED 2019-12-11T17:22:24.943 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:24.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5InvalidShort PASSED 2019-12-11T17:22:28.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:28.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Ivalid STARTED 2019-12-11T17:22:29.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:29.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Ivalid PASSED 2019-12-11T17:22:32.943 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:32.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5None STARTED 2019-12-11T17:22:34.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:34.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5None PASSED 2019-12-11T17:22:37.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:37.044 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Unreadable STARTED 2019-12-11T17:22:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:38.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Unreadable PASSED 2019-12-11T17:22:41.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:41.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectHeadZeroBytes STARTED 2019-12-11T17:22:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:44.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectHeadZeroBytes PASSED 2019-12-11T17:22:51.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:51.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterDot STARTED 2019-12-11T17:22:57.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:22:57.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterDot PASSED 2019-12-11T17:23:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:23:09.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterNotExist STARTED 2019-12-11T17:23:15.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:23:15.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterNotExist PASSED 2019-12-11T17:23:16.815 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:51:03.745 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:16.834 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:51:03.764 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:16.836 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:51:03.766 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:16.860 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:51:03.790 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:16.870 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:51:03.770 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:16.875 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:51:03.776 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:16.919 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:51:03.849 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:16.944 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:51:03.874 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:17.057 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:51:03.957 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:23:24.044 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:23:24.044 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterPercentage STARTED 2019-12-11T17:23:29.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:23:29.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterPercentage PASSED 2019-12-11T17:23:40.044 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:23:40.044 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerAfterList STARTED 2019-12-11T17:23:44.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:23:44.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerAfterList PASSED 2019-12-11T17:23:55.751 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:23:55.751 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerBeforeList STARTED 2019-12-11T17:24:01.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:01.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerBeforeList PASSED 2019-12-11T17:24:12.044 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:12.044 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerNotInList STARTED 2019-12-11T17:24:16.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:16.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerNotInList PASSED 2019-12-11T17:24:25.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:25.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerUnreadable STARTED 2019-12-11T17:24:31.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:31.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMarkerUnreadable PASSED 2019-12-11T17:24:40.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:40.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNegative STARTED 2019-12-11T17:24:45.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:45.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNegative PASSED 2019-12-11T17:24:56.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:24:56.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNone STARTED 2019-12-11T17:25:02.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:02.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNone PASSED 2019-12-11T17:25:07.260 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:52:54.190 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.268 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:52:54.198 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.279 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:52:54.209 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.283 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:52:54.213 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.285 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:52:54.186 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.290 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:52:54.190 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.359 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:52:54.289 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.367 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:52:54.297 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:07.396 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:52:54.296 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:25:09.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:09.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysOne STARTED 2019-12-11T17:25:15.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:15.744 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysOne PASSED 2019-12-11T17:25:23.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:23.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysZero STARTED 2019-12-11T17:25:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:29.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysZero PASSED 2019-12-11T17:25:37.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:37.144 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixAlt STARTED 2019-12-11T17:25:43.453 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:43.453 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixAlt PASSED 2019-12-11T17:25:52.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:52.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixBasic STARTED 2019-12-11T17:25:58.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:25:58.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixBasic PASSED 2019-12-11T17:26:07.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:26:07.450 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterAlt STARTED 2019-12-11T17:26:14.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:26:14.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterAlt PASSED 2019-12-11T17:26:25.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:26:25.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterBasic STARTED 2019-12-11T17:26:33.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:26:33.744 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterBasic PASSED 2019-12-11T17:26:41.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:26:41.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterDelimiterNotExist STARTED 2019-12-11T17:26:47.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:26:47.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterDelimiterNotExist PASSED 2019-12-11T17:26:58.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:26:58.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixDelimiterNotExist STARTED 2019-12-11T17:27:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:27:06.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixDelimiterNotExist PASSED 2019-12-11T17:27:16.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:27:16.744 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixNotExist STARTED 2019-12-11T17:27:23.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:27:23.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixNotExist PASSED 2019-12-11T17:27:34.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:27:34.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixEmpty STARTED 2019-12-11T17:27:40.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:27:40.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixEmpty PASSED 2019-12-11T17:27:49.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:27:49.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNone STARTED 2019-12-11T17:27:57.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:27:57.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNone PASSED 2019-12-11T17:28:08.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:08.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNotExist STARTED 2019-12-11T17:28:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:13.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNotExist PASSED 2019-12-11T17:28:23.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:23.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixUnreadable STARTED 2019-12-11T17:28:29.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:29.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectListPrefixUnreadable PASSED 2019-12-11T17:28:36.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:36.744 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectReadFromNonExistantBucket STARTED 2019-12-11T17:28:39.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:39.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectReadFromNonExistantBucket PASSED 2019-12-11T17:28:44.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:44.444 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectReadNotExist STARTED 2019-12-11T17:28:46.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:46.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectReadNotExist PASSED 2019-12-11T17:28:51.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:51.345 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteCacheControl STARTED 2019-12-11T17:28:55.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:28:55.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteCacheControl PASSED 2019-12-11T17:29:01.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:01.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteCheckEtag STARTED 2019-12-11T17:29:04.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:04.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteCheckEtag PASSED 2019-12-11T17:29:11.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:11.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteReadUpdateReadDelete STARTED 2019-12-11T17:29:18.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:18.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteReadUpdateReadDelete PASSED 2019-12-11T17:29:24.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:24.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteToNonExistantBucket STARTED 2019-12-11T17:29:24.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:24.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testObjectWriteToNonExistantBucket PASSED 2019-12-11T17:29:25.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:25.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testRangedSkipLeadingBytesResponseCode STARTED 2019-12-11T17:29:29.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:29.744 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testRangedSkipLeadingBytesResponseCode PASSED 2019-12-11T17:29:36.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:36.644 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testRangedrequestResponseCode STARTED 2019-12-11T17:29:40.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:40.844 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testRangedrequestResponseCode PASSED 2019-12-11T17:29:45.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:45.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testSSEKMSNoKey STARTED 2019-12-11T17:29:47.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:47.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testSSEKMSNoKey PASSED 2019-12-11T17:29:53.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:53.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testSSEKMSNotDeclared STARTED 2019-12-11T17:29:55.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:29:55.744 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testSSEKMSNotDeclared PASSED 2019-12-11T17:30:02.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:02.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testSSEKMSPresent STARTED 2019-12-11T17:30:05.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:05.344 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testSSEKMSPresent PASSED 2019-12-11T17:30:11.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:11.544 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testUploadFileHLAPIBigFile STARTED 2019-12-11T17:30:19.477 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:58:06.406 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.483 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:58:06.413 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.490 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:58:06.420 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.494 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:58:06.424 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.500 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:58:06.401 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.504 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:58:06.405 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.661 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:58:06.591 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.669 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:58:06.599 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:19.694 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:58:06.595 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:30:21.244 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:21.244 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testUploadFileHLAPIBigFile PASSED 2019-12-11T17:30:27.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:27.944 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testUploadFileListHLAPI STARTED 2019-12-11T17:30:38.044 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:38.044 INFO:teuthology.orchestra.run.smithi079.stdout:suite > Object tests > ObjectTest.testUploadFileListHLAPI PASSED 2019-12-11T17:30:46.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:46.144 INFO:teuthology.orchestra.run.smithi079.stdout:---------------------------------------------------------------------- 2019-12-11T17:30:46.144 INFO:teuthology.orchestra.run.smithi079.stdout:| Results: SUCCESS (72 tests, 72 successes, 0 failures, 0 skipped) | 2019-12-11T17:30:46.144 INFO:teuthology.orchestra.run.smithi079.stdout:---------------------------------------------------------------------- 2019-12-11T17:30:46.177 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:30:46.178 INFO:teuthology.orchestra.run.smithi079.stdout:BUILD SUCCESSFUL in 21m 30s 2019-12-11T17:30:46.178 INFO:teuthology.orchestra.run.smithi079.stdout:5 actionable tasks: 5 executed 2019-12-11T17:30:46.184 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:30:46.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.311+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.337 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.333+0000 7ff24c593ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:30:46.337 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.333+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.342+0000 7ff24c593ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:30:46.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.342+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.347+0000 7ff24c593ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:30:46.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.347+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.359 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.355+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:30:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.553+0000 7ff24c593ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:30:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.553+0000 7ff24c593ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:30:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.553+0000 7ff24c593ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:30:46.558 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.553+0000 7ff24c593ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:30:46.558 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.553+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.562+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:30:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.562+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.586 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.582+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:30:46.586 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.582+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.598 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.594+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:30:46.598 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.594+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.610+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:30:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.610+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.615+0000 7ff24c593ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:30:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.615+0000 7ff24c593ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:30:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.615+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.623 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.619+0000 7ff24c593ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:30:46.623 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.619+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.631 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.627+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:30:46.631 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.627+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.641+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:30:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.641+0000 7ff24c593ec0 20 zonegroup default 2019-12-11T17:30:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.641+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.645+0000 7ff24c593ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:30:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.645+0000 7ff24c593ec0 10 Cannot find current period zone using local zone 2019-12-11T17:30:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.645+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.650+0000 7ff24c593ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:30:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.650+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.657+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:30:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.657+0000 7ff24c593ec0 20 rados->read ofs=0 len=0 2019-12-11T17:30:46.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.680+0000 7ff24c593ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:30:46.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.680+0000 7ff24c593ec0 20 zone default found 2019-12-11T17:30:46.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.680+0000 7ff24c593ec0 20 started sync module instance, tier type = 2019-12-11T17:30:46.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:46.680+0000 7ff24c593ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:30:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.063+0000 7ff24c593ec0 20 add_watcher() i=0 2019-12-11T17:30:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.063+0000 7ff24c593ec0 20 add_watcher() i=1 2019-12-11T17:30:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.063+0000 7ff24c593ec0 20 add_watcher() i=2 2019-12-11T17:30:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.063+0000 7ff24c593ec0 20 add_watcher() i=3 2019-12-11T17:30:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.063+0000 7ff24c593ec0 20 add_watcher() i=4 2019-12-11T17:30:47.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.063+0000 7ff24c593ec0 20 add_watcher() i=5 2019-12-11T17:30:47.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.085+0000 7ff24c593ec0 20 add_watcher() i=6 2019-12-11T17:30:47.096 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.092+0000 7ff24c593ec0 20 add_watcher() i=7 2019-12-11T17:30:47.096 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.092+0000 7ff24c593ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:30:47.097 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.092+0000 7ff209fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:30:47.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.097+0000 7ff2397bd700 0 -- 172.21.15.79:0/1923432445 >> v2:172.21.15.79:3300/0 conn(0x55ca321acd40 msgr2=0x55ca321ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:30:47.141 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:47.137+0000 7ff238fbc700 0 -- 172.21.15.79:0/2175287096 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x55ca321e33c0 msgr2=0x55ca321e57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:30:48.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.771+0000 7ff17bfff700 20 reqs_thread_entry: start 2019-12-11T17:30:48.776 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.771+0000 7ff179ffb700 20 reqs_thread_entry: start 2019-12-11T17:30:48.776 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.771+0000 7ff24c593ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:30:48.776 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.772+0000 7ff167fff700 20 reqs_thread_entry: start 2019-12-11T17:30:48.776 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.772+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:30:48.857 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.853+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:48.888 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.884+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:48.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:48.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1073', time=2019-12-11T19:14:13.632484+0000, chain.objs.size()=9 2019-12-11T17:30:48.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.1 2019-12-11T17:30:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.1_1 2019-12-11T17:30:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.2 2019-12-11T17:30:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.2_1 2019-12-11T17:30:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.3 2019-12-11T17:30:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.3_1 2019-12-11T17:30:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.908+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.4 2019-12-11T17:30:48.914 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.909+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.4_1 2019-12-11T17:30:48.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:48.909+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~4VNErFyyjePyJE0uVdjIjmkI5ZZEncv.5 2019-12-11T17:30:49.286 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.282+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:49.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.368+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:30:49.417 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.411+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:49.429 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.426+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:49.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:49.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1242', time=2019-12-11T19:20:41.171474+0000, chain.objs.size()=7 2019-12-11T17:30:49.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~qhst3YJIhiuo_iRobZJU6VjTo-GLIjE.1 2019-12-11T17:30:49.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~qhst3YJIhiuo_iRobZJU6VjTo-GLIjE.1_1 2019-12-11T17:30:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~qhst3YJIhiuo_iRobZJU6VjTo-GLIjE.1_2 2019-12-11T17:30:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~qhst3YJIhiuo_iRobZJU6VjTo-GLIjE.2 2019-12-11T17:30:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~qhst3YJIhiuo_iRobZJU6VjTo-GLIjE.2_1 2019-12-11T17:30:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~qhst3YJIhiuo_iRobZJU6VjTo-GLIjE.2_2 2019-12-11T17:30:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.437+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~qhst3YJIhiuo_iRobZJU6VjTo-GLIjE.3 2019-12-11T17:30:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.775+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:49.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.889+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:30:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.932+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:49.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.946+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:49.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:49.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1093', time=2019-12-11T19:15:35.563897+0000, chain.objs.size()=7 2019-12-11T17:30:49.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~mFScTDEmboOHK5w2VdXzQJLbEE6ELx2.1 2019-12-11T17:30:49.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~mFScTDEmboOHK5w2VdXzQJLbEE6ELx2.1_1 2019-12-11T17:30:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~mFScTDEmboOHK5w2VdXzQJLbEE6ELx2.1_2 2019-12-11T17:30:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~mFScTDEmboOHK5w2VdXzQJLbEE6ELx2.2 2019-12-11T17:30:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~mFScTDEmboOHK5w2VdXzQJLbEE6ELx2.2_1 2019-12-11T17:30:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~mFScTDEmboOHK5w2VdXzQJLbEE6ELx2.2_2 2019-12-11T17:30:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:49.958+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~mFScTDEmboOHK5w2VdXzQJLbEE6ELx2.3 2019-12-11T17:30:50.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.209+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:50.425 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.421+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:30:50.468 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.464+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:50.482 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.478+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1262', time=2019-12-11T19:21:18.635887+0000, chain.objs.size()=9 2019-12-11T17:30:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__multipart_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.1 2019-12-11T17:30:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__shadow_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.1_1 2019-12-11T17:30:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__multipart_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.2 2019-12-11T17:30:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__shadow_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.2_1 2019-12-11T17:30:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__multipart_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.3 2019-12-11T17:30:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__shadow_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.3_1 2019-12-11T17:30:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__multipart_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.4 2019-12-11T17:30:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__shadow_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.4_1 2019-12-11T17:30:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.491+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.76__multipart_key1.2~RuDPm5bTbWH7pkEbqWX9spprCMaUGL6.5 2019-12-11T17:30:50.886 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.881+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:50.992 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:50.988+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:30:51.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.031+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:51.050 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.046+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:51.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.058+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:51.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.058+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1915', time=2019-12-11T19:30:39.257468+0000, chain.objs.size()=9 2019-12-11T17:30:51.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.058+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__multipart_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.1 2019-12-11T17:30:51.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.058+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__shadow_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.1_1 2019-12-11T17:30:51.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.058+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__multipart_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.2 2019-12-11T17:30:51.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.058+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__shadow_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.2_1 2019-12-11T17:30:51.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.059+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__multipart_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.3 2019-12-11T17:30:51.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.059+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__shadow_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.3_1 2019-12-11T17:30:51.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.059+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__multipart_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.4 2019-12-11T17:30:51.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.059+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__shadow_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.4_1 2019-12-11T17:30:51.064 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.059+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.119__multipart_key1/data/file.mpg.2~1yG8uSt30kUNv3fibYHntgnx8-FJY5C.5 2019-12-11T17:30:51.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.445+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:51.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.604+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:30:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.643+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:51.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.658+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:51.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:51.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1225', time=2019-12-11T19:19:16.302839+0000, chain.objs.size()=9 2019-12-11T17:30:51.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.1 2019-12-11T17:30:51.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.1_1 2019-12-11T17:30:51.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.2 2019-12-11T17:30:51.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.2_1 2019-12-11T17:30:51.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.3 2019-12-11T17:30:51.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.3_1 2019-12-11T17:30:51.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.4 2019-12-11T17:30:51.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.4_1 2019-12-11T17:30:51.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.671+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~QHo7E2N6XlJeYCVc3h2mgp04SdkgOtA.5 2019-12-11T17:30:51.931 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:51.927+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:52.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.059+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:30:52.108 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.104+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:52.130 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.126+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:52.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:30:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1081', time=2019-12-11T19:14:35.298514+0000, chain.objs.size()=9 2019-12-11T17:30:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.1 2019-12-11T17:30:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.1_1 2019-12-11T17:30:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.2 2019-12-11T17:30:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.2_1 2019-12-11T17:30:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.3 2019-12-11T17:30:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.3_1 2019-12-11T17:30:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.4 2019-12-11T17:30:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.4_1 2019-12-11T17:30:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~dPSE9b8NgMMZw9iqy_an6IdkiZyLR2P.5 2019-12-11T17:30:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1237', time=2019-12-11T19:20:21.800851+0000, chain.objs.size()=7 2019-12-11T17:30:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~BHdqtttzCiW_ONKvuisvX0xbHZSEW-d.1 2019-12-11T17:30:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.153+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~BHdqtttzCiW_ONKvuisvX0xbHZSEW-d.1_1 2019-12-11T17:30:52.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.153+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~BHdqtttzCiW_ONKvuisvX0xbHZSEW-d.1_2 2019-12-11T17:30:52.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.352+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~BHdqtttzCiW_ONKvuisvX0xbHZSEW-d.2 2019-12-11T17:30:52.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.516+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~BHdqtttzCiW_ONKvuisvX0xbHZSEW-d.2_1 2019-12-11T17:30:52.521 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.516+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~BHdqtttzCiW_ONKvuisvX0xbHZSEW-d.2_2 2019-12-11T17:30:52.521 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.516+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~BHdqtttzCiW_ONKvuisvX0xbHZSEW-d.3 2019-12-11T17:30:52.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.737+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:52.840 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.836+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:30:52.907 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.903+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:52.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.923+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:52.943 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:52.943 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1247', time=2019-12-11T19:20:44.539034+0000, chain.objs.size()=7 2019-12-11T17:30:52.943 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~6ocOiqNtl8MVc5aTZbZEyYpBnlPIlzQ.1 2019-12-11T17:30:52.943 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~6ocOiqNtl8MVc5aTZbZEyYpBnlPIlzQ.1_1 2019-12-11T17:30:52.943 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~6ocOiqNtl8MVc5aTZbZEyYpBnlPIlzQ.1_2 2019-12-11T17:30:52.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~6ocOiqNtl8MVc5aTZbZEyYpBnlPIlzQ.2 2019-12-11T17:30:52.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~6ocOiqNtl8MVc5aTZbZEyYpBnlPIlzQ.2_1 2019-12-11T17:30:52.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~6ocOiqNtl8MVc5aTZbZEyYpBnlPIlzQ.2_2 2019-12-11T17:30:52.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:52.939+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~6ocOiqNtl8MVc5aTZbZEyYpBnlPIlzQ.3 2019-12-11T17:30:53.295 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.291+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:53.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.397+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:30:53.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.445+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:53.466 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.462+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:53.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:30:53.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1042', time=2019-12-11T19:12:21.362066+0000, chain.objs.size()=9 2019-12-11T17:30:53.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.1 2019-12-11T17:30:53.488 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.1_1 2019-12-11T17:30:53.488 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.2 2019-12-11T17:30:53.488 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.2_1 2019-12-11T17:30:53.488 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.3 2019-12-11T17:30:53.488 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.3_1 2019-12-11T17:30:53.488 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.482+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.4 2019-12-11T17:30:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.483+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.4_1 2019-12-11T17:30:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.483+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.5 2019-12-11T17:30:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.483+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1044', time=2019-12-11T19:12:24.996620+0000, chain.objs.size()=9 2019-12-11T17:30:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.483+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.1 2019-12-11T17:30:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.483+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.1_1 2019-12-11T17:30:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.483+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.2 2019-12-11T17:30:53.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.799+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.2_1 2019-12-11T17:30:53.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.799+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.3 2019-12-11T17:30:53.856 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.852+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.3_1 2019-12-11T17:30:53.856 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.852+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.4 2019-12-11T17:30:53.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.975+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__shadow_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.4_1 2019-12-11T17:30:53.979 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:53.975+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.64__multipart_key1.2~ZbhM9xaAraXWzo660407B69I4hKNouy.5 2019-12-11T17:30:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.244+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:54.438 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.434+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:30:54.492 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.489+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.508+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:54.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.518+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:54.564 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.560+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:30:54.612 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.608+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:54.628 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.624+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:54.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.632+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:54.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.672+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:30:54.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.727+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:54.750 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.746+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1186', time=2019-12-11T19:17:23.841149+0000, chain.objs.size()=9 2019-12-11T17:30:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.1 2019-12-11T17:30:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.1_1 2019-12-11T17:30:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.2 2019-12-11T17:30:54.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.2_1 2019-12-11T17:30:54.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.3 2019-12-11T17:30:54.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.3_1 2019-12-11T17:30:54.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.4 2019-12-11T17:30:54.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.4_1 2019-12-11T17:30:54.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:54.762+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file2.mpg.2~ZWVkAZ7rLgbEFDZxFnshUeM--ruVPmi.5 2019-12-11T17:30:55.022 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.018+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:55.117 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.112+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:30:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.149+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:55.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.163+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:55.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.172+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:55.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.202+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:30:55.249 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.245+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:55.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.264+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:55.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:55.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT', time=2019-12-11T19:10:26.091643+0000, chain.objs.size()=9 2019-12-11T17:30:55.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__multipart_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.1 2019-12-11T17:30:55.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__shadow_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.1_1 2019-12-11T17:30:55.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__multipart_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.2 2019-12-11T17:30:55.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__shadow_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.2_1 2019-12-11T17:30:55.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__multipart_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.3 2019-12-11T17:30:55.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__shadow_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.3_1 2019-12-11T17:30:55.284 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__multipart_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.4 2019-12-11T17:30:55.285 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__shadow_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.4_1 2019-12-11T17:30:55.285 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.279+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.52__multipart_key1.2~Hc1U5bi5iOUvr8yA9hxncqOBkCb2wfT.5 2019-12-11T17:30:55.503 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.499+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:55.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.644+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:30:55.692 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.688+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:55.716 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.702+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:55.717 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.712+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:55.717 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.712+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.973', time=2019-12-11T19:11:09.819489+0000, chain.objs.size()=2 2019-12-11T17:30:55.717 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.712+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.56__shadow_.fLBofHAFv8qlEcTmdFmXUG9wHIexRhd_1 2019-12-11T17:30:55.717 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.712+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.56__shadow_.fLBofHAFv8qlEcTmdFmXUG9wHIexRhd_2 2019-12-11T17:30:55.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.825+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:55.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.948+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:30:55.988 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.984+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:56.002 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:55.998+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:56.033 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2019-12-11T17:30:56.033 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1218', time=2019-12-11T19:18:54.520645+0000, chain.objs.size()=9 2019-12-11T17:30:56.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.1 2019-12-11T17:30:56.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.1_1 2019-12-11T17:30:56.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.2 2019-12-11T17:30:56.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.2_1 2019-12-11T17:30:56.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.3 2019-12-11T17:30:56.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.3_1 2019-12-11T17:30:56.034 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.4 2019-12-11T17:30:56.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.4_1 2019-12-11T17:30:56.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~HHSxZbs_r1cfsbbckRZkY5V2j29Yh1z.5 2019-12-11T17:30:56.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1277', time=2019-12-11T19:21:48.969137+0000, chain.objs.size()=9 2019-12-11T17:30:56.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__multipart_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.1 2019-12-11T17:30:56.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__shadow_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.1_1 2019-12-11T17:30:56.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.029+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__multipart_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.2 2019-12-11T17:30:56.318 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__shadow_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.2_1 2019-12-11T17:30:56.319 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__multipart_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.3 2019-12-11T17:30:56.319 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__shadow_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.3_1 2019-12-11T17:30:56.319 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__multipart_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.4 2019-12-11T17:30:56.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__shadow_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.4_1 2019-12-11T17:30:56.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.77__multipart_key1.2~isYaAcgk-qo4Av2waGa1RCTIBOth-fE.5 2019-12-11T17:30:56.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz', time=2019-12-11T19:22:09.321319+0000, chain.objs.size()=39 2019-12-11T17:30:56.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.1 2019-12-11T17:30:56.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.2 2019-12-11T17:30:56.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.3 2019-12-11T17:30:56.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.4 2019-12-11T17:30:56.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.314+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.5 2019-12-11T17:30:56.546 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.542+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.6 2019-12-11T17:30:56.547 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.542+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.7 2019-12-11T17:30:56.547 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.542+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.8 2019-12-11T17:30:56.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.552+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.9 2019-12-11T17:30:56.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.552+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.10 2019-12-11T17:30:56.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.632+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.11 2019-12-11T17:30:56.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.632+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.12 2019-12-11T17:30:56.636 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.632+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.Ol7Uxccwa8un_2u7o0aLk449iTnscUx.13 2019-12-11T17:30:56.779 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.775+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.14 2019-12-11T17:30:56.779 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.775+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.15 2019-12-11T17:30:56.779 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.775+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.CY1_XwIV4n46k-zDjZ3jkRBwr4Rt6pC.16 2019-12-11T17:30:56.789 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.785+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.8wJVUsAVwt0a-kv7LFDlf_zxy1GmEWA.17 2019-12-11T17:30:56.789 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.785+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.j-oYZx3H7pzptQ940Y73pKGl4dgWxRm.18 2019-12-11T17:30:56.789 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:56.785+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.685LdgBKGzwQRvhrUdhfLjJlnSvVJl6.19 2019-12-11T17:30:57.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.049+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.JOsLKSJEMulyE4MrvlVE3LOP7xHmS3a.20 2019-12-11T17:30:57.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.049+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.21 2019-12-11T17:30:57.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.049+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.22 2019-12-11T17:30:57.054 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.049+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.23 2019-12-11T17:30:57.054 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.049+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.24 2019-12-11T17:30:57.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.187+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.25 2019-12-11T17:30:57.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.187+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.26 2019-12-11T17:30:57.231 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.224+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.27 2019-12-11T17:30:57.231 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.224+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.28 2019-12-11T17:30:57.234 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.230+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.29 2019-12-11T17:30:57.234 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.230+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.30 2019-12-11T17:30:57.235 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.230+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.31 2019-12-11T17:30:57.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.238+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.32 2019-12-11T17:30:57.370 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.366+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.33 2019-12-11T17:30:57.370 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.366+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.34 2019-12-11T17:30:57.370 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.366+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.35 2019-12-11T17:30:57.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.379+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.36 2019-12-11T17:30:57.411 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.407+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.37 2019-12-11T17:30:57.501 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.497+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.38 2019-12-11T17:30:57.502 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.497+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.78__multipart_key1.2~pAASKTKzGv4rfaiO_HlGn9mOvHvykDz.40 2019-12-11T17:30:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.706+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:57.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.805+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:30:57.854 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.850+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:57.877 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.873+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:57.888 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.885+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:57.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:57.931+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:30:58.005 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.002+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:58.033 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.029+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:58.048 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.041+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.080+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:30:58.137 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.133+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:58.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.150+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:58.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.159+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.196+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:30:58.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.272+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.305+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:58.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:58.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1155', time=2019-12-11T19:16:47.927233+0000, chain.objs.size()=9 2019-12-11T17:30:58.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__multipart_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.1 2019-12-11T17:30:58.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__shadow_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.1_1 2019-12-11T17:30:58.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__multipart_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.2 2019-12-11T17:30:58.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__shadow_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.2_1 2019-12-11T17:30:58.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__multipart_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.3 2019-12-11T17:30:58.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__shadow_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.3_1 2019-12-11T17:30:58.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__multipart_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.4 2019-12-11T17:30:58.341 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__shadow_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.4_1 2019-12-11T17:30:58.341 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.332+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.72__multipart_key1.2~n_BC8Rw1-T497EYG2A2Lgf2zX74zbHZ.5 2019-12-11T17:30:58.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.723+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:58.868 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.863+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:30:58.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.913+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.930+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:58.946 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.942+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:58.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.942+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1057', time=2019-12-11T19:15:37.802690+0000, chain.objs.size()=5 2019-12-11T17:30:58.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.942+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.66__shadow_.DhRSouK14yQ7BEworA-mZP3F-cHFB8a_1 2019-12-11T17:30:58.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.942+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.66__shadow_.DhRSouK14yQ7BEworA-mZP3F-cHFB8a_2 2019-12-11T17:30:58.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.942+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.66__shadow_.DhRSouK14yQ7BEworA-mZP3F-cHFB8a_3 2019-12-11T17:30:58.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.942+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.66__shadow_.DhRSouK14yQ7BEworA-mZP3F-cHFB8a_4 2019-12-11T17:30:58.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:58.942+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.66__shadow_.DhRSouK14yQ7BEworA-mZP3F-cHFB8a_5 2019-12-11T17:30:59.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.194+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:30:59.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.342+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:30:59.409 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.398+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:59.419 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.415+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:59.429 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.425+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:59.457 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.453+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:30:59.540 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.536+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:30:59.570 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.565+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:30:59.589 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:30:59.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1187', time=2019-12-11T19:17:20.634676+0000, chain.objs.size()=9 2019-12-11T17:30:59.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.1 2019-12-11T17:30:59.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.1_1 2019-12-11T17:30:59.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.2 2019-12-11T17:30:59.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.2_1 2019-12-11T17:30:59.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.3 2019-12-11T17:30:59.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.3_1 2019-12-11T17:30:59.591 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.4 2019-12-11T17:30:59.591 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__shadow_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.4_1 2019-12-11T17:30:59.591 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.585+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.73__multipart_file.mpg.2~Udph2VHWQsif9YyFpMLANSts9T2evWD.5 2019-12-11T17:30:59.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:30:59.977+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:31:00.083 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.079+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:31:00.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.120+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:00.139 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.136+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:00.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:31:00.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI', time=2019-12-11T19:17:51.840872+0000, chain.objs.size()=9 2019-12-11T17:31:00.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__multipart_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.7 2019-12-11T17:31:00.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__shadow_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.7_1 2019-12-11T17:31:00.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__multipart_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.8 2019-12-11T17:31:00.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__shadow_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.8_1 2019-12-11T17:31:00.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__multipart_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.9 2019-12-11T17:31:00.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__shadow_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.9_1 2019-12-11T17:31:00.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__multipart_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.10 2019-12-11T17:31:00.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__shadow_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.10_1 2019-12-11T17:31:00.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.74__multipart_key1.2~gN3Q_73nKphuKAil_OlJLK1Tqp40DuI.11 2019-12-11T17:31:00.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1899', time=2019-12-11T19:30:22.627939+0000, chain.objs.size()=9 2019-12-11T17:31:00.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__multipart_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.1 2019-12-11T17:31:00.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__shadow_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.1_1 2019-12-11T17:31:00.158 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.152+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__multipart_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.2 2019-12-11T17:31:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.453+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__shadow_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.2_1 2019-12-11T17:31:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.453+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__multipart_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.3 2019-12-11T17:31:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.453+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__shadow_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.3_1 2019-12-11T17:31:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.453+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__multipart_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.4 2019-12-11T17:31:00.458 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.453+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__shadow_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.4_1 2019-12-11T17:31:00.458 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.453+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.118__multipart_key1.2~kuZhyBx8Og9nTimFT7nwZZ7hy_ve6uR.5 2019-12-11T17:31:00.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.703+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:31:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.835+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:31:00.879 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.875+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:00.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.890+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:00.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.912+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:00.952 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.948+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:31:00.994 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:00.990+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:01.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.011+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:01.022 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.018+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:01.065 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.056+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:31:01.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.273+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:01.297 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.292+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:01.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:31:01.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1065', time=2019-12-11T19:13:33.748588+0000, chain.objs.size()=9 2019-12-11T17:31:01.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.1 2019-12-11T17:31:01.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.1_1 2019-12-11T17:31:01.315 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.2 2019-12-11T17:31:01.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.2_1 2019-12-11T17:31:01.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.3 2019-12-11T17:31:01.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.3_1 2019-12-11T17:31:01.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.4 2019-12-11T17:31:01.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.4_1 2019-12-11T17:31:01.316 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.309+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~qtW8-RLdpwI4W3mHsMQi2Bh8pDzHtc9.5 2019-12-11T17:31:01.647 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.642+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:31:01.734 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.729+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:31:01.792 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.787+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:01.809 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.804+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:01.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:31:01.823 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1232', time=2019-12-11T19:19:53.109321+0000, chain.objs.size()=9 2019-12-11T17:31:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.1 2019-12-11T17:31:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.1_1 2019-12-11T17:31:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.2 2019-12-11T17:31:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.2_1 2019-12-11T17:31:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.3 2019-12-11T17:31:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.3_1 2019-12-11T17:31:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.4 2019-12-11T17:31:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__shadow_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.4_1 2019-12-11T17:31:01.826 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:01.817+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.75__multipart_key1.2~v73UszyFkovBkR_6LKWG472fhqdCih7.5 2019-12-11T17:31:02.102 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.097+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:31:02.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.195+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:31:02.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.241+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:02.272 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.267+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:02.291 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-11T17:31:02.292 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1087', time=2019-12-11T19:14:57.753778+0000, chain.objs.size()=7 2019-12-11T17:31:02.292 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~NXxv8WSI9WiDVPrP7lOS8jRguZWWHzp.1 2019-12-11T17:31:02.292 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~NXxv8WSI9WiDVPrP7lOS8jRguZWWHzp.1_1 2019-12-11T17:31:02.292 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~NXxv8WSI9WiDVPrP7lOS8jRguZWWHzp.1_2 2019-12-11T17:31:02.293 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~NXxv8WSI9WiDVPrP7lOS8jRguZWWHzp.2 2019-12-11T17:31:02.293 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~NXxv8WSI9WiDVPrP7lOS8jRguZWWHzp.2_1 2019-12-11T17:31:02.293 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~NXxv8WSI9WiDVPrP7lOS8jRguZWWHzp.2_2 2019-12-11T17:31:02.293 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~NXxv8WSI9WiDVPrP7lOS8jRguZWWHzp.3 2019-12-11T17:31:02.294 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1136', time=2019-12-11T19:16:22.698077+0000, chain.objs.size()=9 2019-12-11T17:31:02.294 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__multipart_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.1 2019-12-11T17:31:02.294 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__shadow_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.1_1 2019-12-11T17:31:02.294 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__multipart_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.2 2019-12-11T17:31:02.295 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__shadow_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.2_1 2019-12-11T17:31:02.295 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.286+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__multipart_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.3 2019-12-11T17:31:02.586 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.581+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__shadow_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.3_1 2019-12-11T17:31:02.644 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.639+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__multipart_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.4 2019-12-11T17:31:02.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.639+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__shadow_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.4_1 2019-12-11T17:31:02.645 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.639+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.71__multipart_key1.2~D6oivCkAgZ_vkpDAP9umwGjsiffRhsN.5 2019-12-11T17:31:02.933 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:02.928+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:31:03.035 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.030+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:31:03.079 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.074+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:03.094 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.089+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-11T17:31:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 20 garbage collection: RGWGC::process iterating over entry tag='14af9c21-7f56-4de3-9a52-895b5fee19a0.4518.1099', time=2019-12-11T19:15:41.509277+0000, chain.objs.size()=7 2019-12-11T17:31:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~rOg_R6rcsRdy-HtLlRaLBhzK6ydLP-a.1 2019-12-11T17:31:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~rOg_R6rcsRdy-HtLlRaLBhzK6ydLP-a.1_1 2019-12-11T17:31:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~rOg_R6rcsRdy-HtLlRaLBhzK6ydLP-a.1_2 2019-12-11T17:31:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~rOg_R6rcsRdy-HtLlRaLBhzK6ydLP-a.2 2019-12-11T17:31:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~rOg_R6rcsRdy-HtLlRaLBhzK6ydLP-a.2_1 2019-12-11T17:31:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__shadow_key1.2~rOg_R6rcsRdy-HtLlRaLBhzK6ydLP-a.2_2 2019-12-11T17:31:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.101+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.67__multipart_key1.2~rOg_R6rcsRdy-HtLlRaLBhzK6ydLP-a.3 2019-12-11T17:31:03.348 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.343+0000 7ff24c593ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-11T17:31:03.439 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.434+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:31:03.483 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.478+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:03.502 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.497+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:03.511 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.506+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:03.552 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.547+0000 7ff24c593ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:31:03.631 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.626+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:03.651 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.646+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:03.659 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.654+0000 7ff24c593ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.755+0000 7ff24c593ec0 20 remove_watcher() i=0 2019-12-11T17:31:03.761 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.755+0000 7ff24c593ec0 2 removed watcher, disabling cache 2019-12-11T17:31:03.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.806+0000 7ff24c593ec0 20 remove_watcher() i=1 2019-12-11T17:31:03.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.838+0000 7ff24c593ec0 20 remove_watcher() i=2 2019-12-11T17:31:03.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.918+0000 7ff24c593ec0 20 remove_watcher() i=3 2019-12-11T17:31:03.990 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:03.985+0000 7ff24c593ec0 20 remove_watcher() i=4 2019-12-11T17:31:04.031 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.026+0000 7ff24c593ec0 20 remove_watcher() i=5 2019-12-11T17:31:04.074 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.069+0000 7ff24c593ec0 20 remove_watcher() i=6 2019-12-11T17:31:04.114 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.109+0000 7ff24c593ec0 20 remove_watcher() i=7 2019-12-11T17:31:04.124 INFO:teuthology.orchestra.run.smithi079:> radosgw-admin gc process --include-all 2019-12-11T17:31:04.233 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.228+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.255 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.250+0000 7f1549441ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:04.255 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.250+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.258+0000 7f1549441ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:04.263 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.258+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.266+0000 7f1549441ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.266+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.278 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.273+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:04.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.448+0000 7f1549441ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:31:04.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.448+0000 7f1549441ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:31:04.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.448+0000 7f1549441ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:31:04.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.448+0000 7f1549441ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:31:04.454 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.448+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.467 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.462+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:04.467 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.462+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.479+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:04.484 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.479+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.494 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.489+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:04.494 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.489+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.519 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.514+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:04.519 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.514+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.520+0000 7f1549441ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:04.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.520+0000 7f1549441ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:31:04.525 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.520+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.524+0000 7f1549441ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:04.530 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.524+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.537 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.533+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:04.538 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.533+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.551+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.551+0000 7f1549441ec0 20 zonegroup default 2019-12-11T17:31:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.551+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.557+0000 7f1549441ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:04.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.557+0000 7f1549441ec0 10 Cannot find current period zone using local zone 2019-12-11T17:31:04.562 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.557+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.561+0000 7f1549441ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:04.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.561+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.573+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:04.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.573+0000 7f1549441ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:04.595 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.590+0000 7f1549441ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:04.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.590+0000 7f1549441ec0 20 zone default found 2019-12-11T17:31:04.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.590+0000 7f1549441ec0 20 started sync module instance, tier type = 2019-12-11T17:31:04.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.590+0000 7f1549441ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:31:05.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.998+0000 7f1549441ec0 20 add_watcher() i=0 2019-12-11T17:31:05.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.998+0000 7f1549441ec0 20 add_watcher() i=1 2019-12-11T17:31:05.003 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.998+0000 7f1549441ec0 20 add_watcher() i=2 2019-12-11T17:31:05.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.998+0000 7f1549441ec0 20 add_watcher() i=3 2019-12-11T17:31:05.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.998+0000 7f1549441ec0 20 add_watcher() i=4 2019-12-11T17:31:05.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.998+0000 7f1549441ec0 20 add_watcher() i=5 2019-12-11T17:31:05.004 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:04.998+0000 7f1549441ec0 20 add_watcher() i=6 2019-12-11T17:31:05.040 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:05.035+0000 7f1549441ec0 20 add_watcher() i=7 2019-12-11T17:31:05.040 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:05.035+0000 7f1549441ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:31:05.041 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:05.036+0000 7f1506fdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:31:05.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:05.042+0000 7f152ffff700 0 -- 172.21.15.79:0/2080727994 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x562b260afe70 msgr2=0x562b2608fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:31:05.100 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:05.095+0000 7f153666b700 0 -- 172.21.15.79:0/1526438456 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7f1478022480 msgr2=0x7f1478024990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:06.474 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.469+0000 7f14757fa700 20 reqs_thread_entry: start 2019-12-11T17:31:06.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.469+0000 7f146effd700 20 reqs_thread_entry: start 2019-12-11T17:31:06.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.469+0000 7f1549441ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:31:06.475 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.470+0000 7f146d7fa700 20 reqs_thread_entry: start 2019-12-11T17:31:06.476 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.470+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-11T17:31:06.543 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.538+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.566 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.561+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:06.583 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.578+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.627 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.622+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-11T17:31:06.674 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.669+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.683+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:06.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.693+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.726+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-11T17:31:06.778 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.773+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.795 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.790+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:06.804 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.799+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.834 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.830+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-11T17:31:06.887 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.881+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.902 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.897+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:06.910 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.905+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:06.950 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:06.944+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-11T17:31:07.027 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.022+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.051 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.046+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:07.066 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.061+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.111+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-11T17:31:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.190+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.215 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.210+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:07.227 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.221+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.267 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.262+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-11T17:31:07.349 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.344+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.375+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:07.397 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.392+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.448 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.443+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-11T17:31:07.493 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.488+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.504+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:07.518 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.513+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.544+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-11T17:31:07.590 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.585+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.603+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:07.620 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.615+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.643+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-11T17:31:07.732 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.727+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.765 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.760+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.780+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.854 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.848+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-11T17:31:07.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.907+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:07.937 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.933+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:07.948 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:07.944+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.007+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-11T17:31:08.083 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.079+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.101 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.097+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:08.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.108+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.163+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-11T17:31:08.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.205+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.238 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.234+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:08.251 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.247+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.287 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.281+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-11T17:31:08.329 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.325+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.348 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.344+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:08.358 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.354+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.397 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.393+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-11T17:31:08.445 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.441+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.461+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:08.497 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.492+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.537 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.532+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-11T17:31:08.584 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.581+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.606 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.602+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:08.626 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.623+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.666 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.662+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-11T17:31:08.714 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.710+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.739 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.735+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:08.748 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.744+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.788 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.784+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-11T17:31:08.877 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.873+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.909 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.904+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:08.925 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.921+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:08.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:08.972+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-11T17:31:09.026 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.022+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.045 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.041+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:09.054 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.050+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.256 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.252+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-11T17:31:09.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.316+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.342 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.337+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:09.354 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.349+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.402 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.397+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-11T17:31:09.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.455+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.478 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.473+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:09.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.481+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.517 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.513+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-11T17:31:09.563 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.559+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.574+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:09.588 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.584+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.618 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.614+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-11T17:31:09.662 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.658+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.676+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:09.690 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.687+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.722+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-11T17:31:09.765 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.761+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.781 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.777+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:09.790 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.786+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.835 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.831+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-11T17:31:09.874 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.871+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.889 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.886+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:09.897 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.893+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:09.931 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:09.927+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-11T17:31:10.005 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.001+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.040 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.036+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:10.058 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.054+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.117 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.113+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-11T17:31:10.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.173+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.196+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:10.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.207+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.271+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-11T17:31:10.340 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.336+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.369 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.365+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:10.383 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.379+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.431+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-11T17:31:10.514 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.510+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.540 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.536+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:10.560 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.556+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.590+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-11T17:31:10.677 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.673+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.692+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:10.705 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.701+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.806 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.802+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-11T17:31:10.851 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.847+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.868 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.864+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:10.875 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.871+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.913 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.909+0000 7f1549441ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-11T17:31:10.958 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.954+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:10.973 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.969+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-11T17:31:10.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:10.977+0000 7f1549441ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-11T17:31:11.095 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.091+0000 7f1549441ec0 20 remove_watcher() i=0 2019-12-11T17:31:11.095 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.091+0000 7f1549441ec0 2 removed watcher, disabling cache 2019-12-11T17:31:11.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.151+0000 7f1549441ec0 20 remove_watcher() i=1 2019-12-11T17:31:11.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.192+0000 7f1549441ec0 20 remove_watcher() i=2 2019-12-11T17:31:11.272 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.268+0000 7f1549441ec0 20 remove_watcher() i=3 2019-12-11T17:31:11.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.313+0000 7f1549441ec0 20 remove_watcher() i=4 2019-12-11T17:31:11.360 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.356+0000 7f1549441ec0 20 remove_watcher() i=5 2019-12-11T17:31:11.425 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.421+0000 7f1549441ec0 20 remove_watcher() i=6 2019-12-11T17:31:11.469 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:11.464+0000 7f1549441ec0 20 remove_watcher() i=7 2019-12-11T17:31:11.475 INFO:teuthology.run_tasks:Running task s3tests... 2019-12-11T17:31:11.498 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'rgw_server': 'client.0', 'force-branch': 'public-access-qa', 'git_remote': 'git://github.com/theanalyst/', 'slow_backend': True}} 2019-12-11T17:31:11.498 INFO:tasks.s3tests:Downloading s3-tests... 2019-12-11T17:31:11.499 INFO:tasks.s3tests:Using branch 'public-access-qa' for s3tests 2019-12-11T17:31:11.499 INFO:teuthology.orchestra.run.smithi079:> git clone -b public-access-qa git://github.com/theanalyst/s3-tests.git /home/ubuntu/cephtest/s3-tests 2019-12-11T17:31:11.647 INFO:teuthology.orchestra.run.smithi079.stdout:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2019-12-11T17:31:11.930 INFO:tasks.s3tests:Creating rgw users... 2019-12-11T17:31:11.930 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2019-12-11T17:31:11.931 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key SYWZYEEVOCKRYXJJFPLE --secret jO3MiwHVgQt1wlHkIe0FS0oFo3vTdza8EzI1+/qTt6ZvB+F1HyOVwg== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2019-12-11T17:31:12.047 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.043+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.069 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.065+0000 7fb89cc0cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:12.070 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.065+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.077 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.073+0000 7fb89cc0cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:12.077 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.073+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.082 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.078+0000 7fb89cc0cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:12.082 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.078+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.085+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:12.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.277+0000 7fb89cc0cec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:31:12.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.277+0000 7fb89cc0cec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:31:12.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.277+0000 7fb89cc0cec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:31:12.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.277+0000 7fb89cc0cec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:31:12.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.277+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.298 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.292+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:12.298 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.292+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.312+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:12.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.313+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.330+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:12.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.330+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.350 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.345+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:12.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.345+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.351+0000 7fb89cc0cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:12.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.351+0000 7fb89cc0cec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:31:12.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.351+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.363 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.358+0000 7fb89cc0cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:12.364 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.358+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.368+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:12.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.368+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.385+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:12.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.385+0000 7fb89cc0cec0 20 zonegroup default 2019-12-11T17:31:12.389 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.385+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.391+0000 7fb89cc0cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:12.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.391+0000 7fb89cc0cec0 10 Cannot find current period zone using local zone 2019-12-11T17:31:12.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.391+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.397+0000 7fb89cc0cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:12.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.397+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.410 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.406+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:12.411 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.406+0000 7fb89cc0cec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:12.430 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.426+0000 7fb89cc0cec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:12.431 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.426+0000 7fb89cc0cec0 20 zone default found 2019-12-11T17:31:12.431 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.426+0000 7fb89cc0cec0 20 started sync module instance, tier type = 2019-12-11T17:31:12.431 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.426+0000 7fb89cc0cec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:31:12.871 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.867+0000 7fb89cc0cec0 20 add_watcher() i=0 2019-12-11T17:31:12.871 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.867+0000 7fb89cc0cec0 20 add_watcher() i=1 2019-12-11T17:31:12.871 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.867+0000 7fb89cc0cec0 20 add_watcher() i=2 2019-12-11T17:31:12.871 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.867+0000 7fb89cc0cec0 20 add_watcher() i=3 2019-12-11T17:31:12.872 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.867+0000 7fb89cc0cec0 20 add_watcher() i=4 2019-12-11T17:31:12.872 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.867+0000 7fb89cc0cec0 20 add_watcher() i=5 2019-12-11T17:31:12.872 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.867+0000 7fb89cc0cec0 20 add_watcher() i=6 2019-12-11T17:31:12.908 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.904+0000 7fb89cc0cec0 20 add_watcher() i=7 2019-12-11T17:31:12.908 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.904+0000 7fb89cc0cec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:31:12.908 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.904+0000 7fb85a7dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:31:12.911 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.907+0000 7fb889e36700 0 -- 172.21.15.79:0/3238541214 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x55cc72084110 msgr2=0x55cc72086570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:31:12.951 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:12.947+0000 7fb889635700 0 -- 172.21.15.79:0/1911927960 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x55cc720991f0 msgr2=0x55cc7209b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:14.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.573+0000 7fb7d0ff9700 20 reqs_thread_entry: start 2019-12-11T17:31:14.577 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.573+0000 7fb7ca7fc700 20 reqs_thread_entry: start 2019-12-11T17:31:14.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.573+0000 7fb89cc0cec0 20 init_complete bucket index max shards: 0 2019-12-11T17:31:14.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.574+0000 7fb7c8ff9700 20 reqs_thread_entry: start 2019-12-11T17:31:14.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.574+0000 7fb89cc0cec0 20 get_system_obj_state: rctx=0x7ffdf48c0b78 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55cc72105ca0 s->prefetch_data=0 2019-12-11T17:31:14.578 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.574+0000 7fb89cc0cec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-11T17:31:14.587 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.583+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2019-12-11T17:31:14.587 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.583+0000 7fb89cc0cec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T17:31:14.587 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.583+0000 7fb89cc0cec0 20 get_system_obj_state: rctx=0x7ffdf48c0b88 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55cc72107a30 s->prefetch_data=0 2019-12-11T17:31:14.587 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.583+0000 7fb89cc0cec0 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2019-12-11T17:31:14.593 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.589+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2019-12-11T17:31:14.593 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.589+0000 7fb89cc0cec0 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-11T17:31:14.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.589+0000 7fb89cc0cec0 20 get_system_obj_state: rctx=0x7ffdf48c0b88 obj=default.rgw.meta:users.keys:SYWZYEEVOCKRYXJJFPLE state=0x55cc72107a30 s->prefetch_data=0 2019-12-11T17:31:14.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.589+0000 7fb89cc0cec0 10 cache get: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE : miss 2019-12-11T17:31:14.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.596+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE info.flags=0x0 2019-12-11T17:31:14.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.596+0000 7fb89cc0cec0 10 adding default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE to cache LRU end 2019-12-11T17:31:14.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.596+0000 7fb89cc0cec0 20 get_system_obj_state: rctx=0x7ffdf48c0798 obj=default.rgw.meta:users.keys:SYWZYEEVOCKRYXJJFPLE state=0x55cc72107a30 s->prefetch_data=0 2019-12-11T17:31:14.600 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.596+0000 7fb89cc0cec0 10 cache get: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE : type miss (requested=0x6, cached=0x0) 2019-12-11T17:31:14.610 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.606+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE info.flags=0x0 2019-12-11T17:31:14.611 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.606+0000 7fb89cc0cec0 10 moving default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE to cache LRU end 2019-12-11T17:31:14.611 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.606+0000 7fb89cc0cec0 20 get_system_obj_state: rctx=0x7ffdf48c0918 obj=default.rgw.meta:users.keys:SYWZYEEVOCKRYXJJFPLE state=0x55cc72107a30 s->prefetch_data=0 2019-12-11T17:31:14.611 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.606+0000 7fb89cc0cec0 10 cache get: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE : type miss (requested=0x6, cached=0x0) 2019-12-11T17:31:14.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.614+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE info.flags=0x0 2019-12-11T17:31:14.618 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.614+0000 7fb89cc0cec0 10 moving default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE to cache LRU end 2019-12-11T17:31:14.672 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.668+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T17:31:14.673 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.668+0000 7fb89cc0cec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T17:31:14.673 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.668+0000 7fb89cc0cec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=636 2019-12-11T17:31:14.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.679+0000 7fb86affd700 10 RGWWatcher::handle_notify() notify_id 579820585000 cookie 94336574741072 notifier 4592 bl.length()=636 2019-12-11T17:31:14.683 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.679+0000 7fb86affd700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T17:31:14.684 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.679+0000 7fb86affd700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T17:31:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.961+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-11T17:31:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.961+0000 7fb89cc0cec0 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-11T17:31:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.961+0000 7fb89cc0cec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=194 2019-12-11T17:31:14.974 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.970+0000 7fb86affd700 10 RGWWatcher::handle_notify() notify_id 579820585001 cookie 94336574741072 notifier 4592 bl.length()=194 2019-12-11T17:31:14.974 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.970+0000 7fb86affd700 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-11T17:31:14.974 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:14.970+0000 7fb86affd700 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-11T17:31:15.558 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:15.554+0000 7fb89cc0cec0 10 cache put: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE info.flags=0x7 2019-12-11T17:31:15.558 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:15.554+0000 7fb89cc0cec0 10 moving default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE to cache LRU end 2019-12-11T17:31:15.558 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:15.554+0000 7fb89cc0cec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=186 2019-12-11T17:31:15.575 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:15.572+0000 7fb86affd700 10 RGWWatcher::handle_notify() notify_id 579820584986 cookie 94336574746480 notifier 4592 bl.length()=186 2019-12-11T17:31:15.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:15.572+0000 7fb86affd700 10 cache put: name=default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE info.flags=0x7 2019-12-11T17:31:15.576 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:15.572+0000 7fb86affd700 10 moving default.rgw.meta+users.keys+SYWZYEEVOCKRYXJJFPLE to cache LRU end 2019-12-11T17:31:16.044 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T17:31:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "bar.client.0", 2019-12-11T17:31:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Mr. bar.client.0", 2019-12-11T17:31:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "bar.client.0+test@test.test", 2019-12-11T17:31:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T17:31:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T17:31:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [], 2019-12-11T17:31:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [ 2019-12-11T17:31:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T17:31:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "bar.client.0", 2019-12-11T17:31:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "access_key": "SYWZYEEVOCKRYXJJFPLE", 2019-12-11T17:31:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "jO3MiwHVgQt1wlHkIe0FS0oFo3vTdza8EzI1+/qTt6ZvB+F1HyOVwg==" 2019-12-11T17:31:16.046 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T17:31:16.046 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T17:31:16.046 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [], 2019-12-11T17:31:16.046 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [ 2019-12-11T17:31:16.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T17:31:16.046 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "user-policy", 2019-12-11T17:31:16.047 INFO:teuthology.orchestra.run.smithi079.stdout: "perm": "*" 2019-12-11T17:31:16.047 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T17:31:16.048 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T17:31:16.048 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T17:31:16.048 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T17:31:16.048 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T17:31:16.048 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T17:31:16.048 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T17:31:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T17:31:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T17:31:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T17:31:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T17:31:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T17:31:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T17:31:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T17:31:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T17:31:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T17:31:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T17:31:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T17:31:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T17:31:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T17:31:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T17:31:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T17:31:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T17:31:16.051 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T17:31:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:16.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.106+0000 7fb89cc0cec0 20 remove_watcher() i=0 2019-12-11T17:31:16.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.106+0000 7fb89cc0cec0 2 removed watcher, disabling cache 2019-12-11T17:31:16.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.191+0000 7fb89cc0cec0 20 remove_watcher() i=1 2019-12-11T17:31:16.237 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.233+0000 7fb89cc0cec0 20 remove_watcher() i=2 2019-12-11T17:31:16.309 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.305+0000 7fb89cc0cec0 20 remove_watcher() i=3 2019-12-11T17:31:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.359+0000 7fb89cc0cec0 20 remove_watcher() i=4 2019-12-11T17:31:16.421 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.417+0000 7fb89cc0cec0 20 remove_watcher() i=5 2019-12-11T17:31:16.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.486+0000 7fb89cc0cec0 20 remove_watcher() i=6 2019-12-11T17:31:16.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.545+0000 7fb89cc0cec0 20 remove_watcher() i=7 2019-12-11T17:31:16.558 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 2209724815 --totp-seed EIKDY4LK6NKXTGNDODFAV7OMR2LCRQU3LQV5YTJDFVLCHSOPFSHYMEWBEMKV2DYG --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2019-12-11T17:31:16.694 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.690+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.713+0000 7f33bc9b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.713+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.722+0000 7f33bc9b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:16.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.723+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.728+0000 7f33bc9b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:16.732 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.728+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.739 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.736+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:16.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.930+0000 7f33bc9b5ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:31:16.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.930+0000 7f33bc9b5ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:31:16.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.930+0000 7f33bc9b5ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:31:16.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.930+0000 7f33bc9b5ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:31:16.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.930+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.940+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:16.944 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.940+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.962 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.958+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:16.963 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.958+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.967+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:16.972 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.967+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.977+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:16.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.977+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.986 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.982+0000 7f33bc9b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:16.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.982+0000 7f33bc9b5ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:31:16.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.983+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.990 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.986+0000 7f33bc9b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:16.990 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.986+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:16.999 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.995+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:16.999 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:16.995+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:17.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.003+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:17.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.003+0000 7f33bc9b5ec0 20 zonegroup default 2019-12-11T17:31:17.007 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.003+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:17.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.007+0000 7f33bc9b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:17.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.007+0000 7f33bc9b5ec0 10 Cannot find current period zone using local zone 2019-12-11T17:31:17.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.007+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:17.016 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.012+0000 7f33bc9b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:17.016 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.012+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:17.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.019+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:17.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.019+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:17.039 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.035+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:17.039 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.035+0000 7f33bc9b5ec0 20 zone default found 2019-12-11T17:31:17.039 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.035+0000 7f33bc9b5ec0 20 started sync module instance, tier type = 2019-12-11T17:31:17.040 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.035+0000 7f33bc9b5ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:31:17.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.368+0000 7f33bc9b5ec0 20 add_watcher() i=0 2019-12-11T17:31:17.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.368+0000 7f33bc9b5ec0 20 add_watcher() i=1 2019-12-11T17:31:17.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.368+0000 7f33bc9b5ec0 20 add_watcher() i=2 2019-12-11T17:31:17.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.368+0000 7f33bc9b5ec0 20 add_watcher() i=3 2019-12-11T17:31:17.372 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.368+0000 7f33bc9b5ec0 20 add_watcher() i=4 2019-12-11T17:31:17.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.396+0000 7f33bc9b5ec0 20 add_watcher() i=5 2019-12-11T17:31:17.400 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.396+0000 7f33bc9b5ec0 20 add_watcher() i=6 2019-12-11T17:31:17.412 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.408+0000 7f33bc9b5ec0 20 add_watcher() i=7 2019-12-11T17:31:17.412 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.408+0000 7f33bc9b5ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:31:17.412 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.408+0000 7f337a7dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:31:17.440 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.436+0000 7f33aa3e0700 0 -- 172.21.15.79:0/4115506297 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561b6fa5a910 msgr2=0x561b6fa5ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:17.506 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:17.502+0000 7f33aa3e0700 0 -- 172.21.15.79:0/491160796 >> [v2:172.21.15.79:6818/10237,v1:172.21.15.79:6819/10237] conn(0x561b6fa75fc0 msgr2=0x561b6fa783e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:19.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.243+0000 7f32f0ff9700 20 reqs_thread_entry: start 2019-12-11T17:31:19.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.243+0000 7f32ea7fc700 20 reqs_thread_entry: start 2019-12-11T17:31:19.247 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.243+0000 7f33bc9b5ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:31:19.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.243+0000 7f32e8ff9700 20 reqs_thread_entry: start 2019-12-11T17:31:19.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.243+0000 7f33bc9b5ec0 20 get_system_obj_state: rctx=0x7ffdfd68ddf8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x561b6fadfa30 s->prefetch_data=0 2019-12-11T17:31:19.248 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.243+0000 7f33bc9b5ec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-11T17:31:19.267 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.263+0000 7f33bc9b5ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2019-12-11T17:31:19.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.263+0000 7f33bc9b5ec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T17:31:19.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.263+0000 7f33bc9b5ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T17:31:19.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.263+0000 7f33bc9b5ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T17:31:19.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.263+0000 7f33bc9b5ec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T17:31:19.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.263+0000 7f33bc9b5ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.278+0000 7f33bc9b5ec0 20 rados_obj.operate() r=0 bl.length=451 2019-12-11T17:31:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.278+0000 7f33bc9b5ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2019-12-11T17:31:19.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:19.278+0000 7f33bc9b5ec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T17:31:21.364 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.360+0000 7f33bc9b5ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T17:31:21.365 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.360+0000 7f33bc9b5ec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T17:31:21.365 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.360+0000 7f33bc9b5ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=650 2019-12-11T17:31:21.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.373+0000 7f338affd700 10 RGWWatcher::handle_notify() notify_id 588410519594 cookie 94675837137920 notifier 4604 bl.length()=650 2019-12-11T17:31:21.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.373+0000 7f338affd700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T17:31:21.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.373+0000 7f338affd700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T17:31:21.633 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.629+0000 7f33bc9b5ec0 20 remove_watcher() i=0 2019-12-11T17:31:21.633 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.629+0000 7f33bc9b5ec0 2 removed watcher, disabling cache 2019-12-11T17:31:21.671 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.667+0000 7f33bc9b5ec0 20 remove_watcher() i=1 2019-12-11T17:31:21.709 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.705+0000 7f33bc9b5ec0 20 remove_watcher() i=2 2019-12-11T17:31:21.790 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.786+0000 7f33bc9b5ec0 20 remove_watcher() i=3 2019-12-11T17:31:21.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.825+0000 7f33bc9b5ec0 20 remove_watcher() i=4 2019-12-11T17:31:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:21.875+0000 7f33bc9b5ec0 20 remove_watcher() i=5 2019-12-11T17:31:22.554 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:22.550+0000 7f33bc9b5ec0 20 remove_watcher() i=6 2019-12-11T17:31:22.678 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:22.674+0000 7f33bc9b5ec0 20 remove_watcher() i=7 2019-12-11T17:31:22.684 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2019-12-11T17:31:22.684 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key KYHHQNSRBQRKUNJKBGDL --secret O/NJcEIJg1J4PeeQlnizdn2LBCPOVJHYcAWZGpS5U9dFB3lUr3zriA== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2019-12-11T17:31:22.816 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:22.812+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:22.932 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:22.928+0000 7f7282026ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:22.933 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:22.928+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.082 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.078+0000 7f7282026ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:23.083 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.078+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.087 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.083+0000 7f7282026ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:23.087 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.083+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.094 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.090+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:23.268 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.264+0000 7f7282026ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:31:23.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.264+0000 7f7282026ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:31:23.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.264+0000 7f7282026ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:31:23.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.264+0000 7f7282026ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:31:23.269 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.264+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.272+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:23.276 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.272+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.290 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.286+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:23.290 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.286+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.299 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.293+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:23.300 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.293+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.314 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.310+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:23.314 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.310+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.317+0000 7f7282026ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:23.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.317+0000 7f7282026ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:31:23.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.317+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.327 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.323+0000 7f7282026ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:23.327 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.323+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.338 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.335+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:23.339 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.335+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.347+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:23.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.347+0000 7f7282026ec0 20 zonegroup default 2019-12-11T17:31:23.351 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.347+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.353+0000 7f7282026ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:23.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.353+0000 7f7282026ec0 10 Cannot find current period zone using local zone 2019-12-11T17:31:23.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.353+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.358+0000 7f7282026ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:23.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.358+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.373 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.369+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:23.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.369+0000 7f7282026ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:23.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.390+0000 7f7282026ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:23.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.390+0000 7f7282026ec0 20 zone default found 2019-12-11T17:31:23.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.390+0000 7f7282026ec0 20 started sync module instance, tier type = 2019-12-11T17:31:23.395 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.390+0000 7f7282026ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:31:23.884 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.880+0000 7f7282026ec0 20 add_watcher() i=0 2019-12-11T17:31:23.884 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.880+0000 7f7282026ec0 20 add_watcher() i=1 2019-12-11T17:31:23.884 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.880+0000 7f7282026ec0 20 add_watcher() i=2 2019-12-11T17:31:23.884 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.880+0000 7f7282026ec0 20 add_watcher() i=3 2019-12-11T17:31:23.884 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.880+0000 7f7282026ec0 20 add_watcher() i=4 2019-12-11T17:31:23.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.880+0000 7f7282026ec0 20 add_watcher() i=5 2019-12-11T17:31:23.885 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.880+0000 7f7282026ec0 20 add_watcher() i=6 2019-12-11T17:31:23.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.912+0000 7f7282026ec0 20 add_watcher() i=7 2019-12-11T17:31:23.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.912+0000 7f7282026ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:31:23.917 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.913+0000 7f723ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:31:23.923 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:23.919+0000 7f726fa51700 0 -- 172.21.15.79:0/2467231709 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x556da659e720 msgr2=0x556da659eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:31:24.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:24.122+0000 7f726fa51700 0 --2- 172.21.15.79:0/2467231709 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x556da659e720 0x556da659eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-11T17:31:24.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:24.198+0000 7f726fa51700 0 -- 172.21.15.79:0/1879296286 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x556da65a4cb0 msgr2=0x556da65a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:25.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.341+0000 7f71ae7fc700 20 reqs_thread_entry: start 2019-12-11T17:31:25.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.342+0000 7f71a7fff700 20 reqs_thread_entry: start 2019-12-11T17:31:25.346 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.342+0000 7f7282026ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:31:25.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.342+0000 7f71a67fc700 20 reqs_thread_entry: start 2019-12-11T17:31:25.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.342+0000 7f7282026ec0 20 get_system_obj_state: rctx=0x7ffd7c33db08 obj=default.rgw.meta:users.uid:foo.client.0 state=0x556da6620bb0 s->prefetch_data=0 2019-12-11T17:31:25.347 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.342+0000 7f7282026ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-11T17:31:25.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.353+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2019-12-11T17:31:25.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.353+0000 7f7282026ec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T17:31:25.357 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.353+0000 7f7282026ec0 20 get_system_obj_state: rctx=0x7ffd7c33db18 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x556da6622940 s->prefetch_data=0 2019-12-11T17:31:25.358 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.353+0000 7f7282026ec0 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2019-12-11T17:31:25.363 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.359+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2019-12-11T17:31:25.363 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.359+0000 7f7282026ec0 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-11T17:31:25.363 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.359+0000 7f7282026ec0 20 get_system_obj_state: rctx=0x7ffd7c33db18 obj=default.rgw.meta:users.keys:KYHHQNSRBQRKUNJKBGDL state=0x556da6622940 s->prefetch_data=0 2019-12-11T17:31:25.364 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.359+0000 7f7282026ec0 10 cache get: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL : miss 2019-12-11T17:31:25.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.364+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL info.flags=0x0 2019-12-11T17:31:25.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.364+0000 7f7282026ec0 10 adding default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL to cache LRU end 2019-12-11T17:31:25.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.364+0000 7f7282026ec0 20 get_system_obj_state: rctx=0x7ffd7c33d728 obj=default.rgw.meta:users.keys:KYHHQNSRBQRKUNJKBGDL state=0x556da6622940 s->prefetch_data=0 2019-12-11T17:31:25.369 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.364+0000 7f7282026ec0 10 cache get: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL : type miss (requested=0x6, cached=0x0) 2019-12-11T17:31:25.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.370+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL info.flags=0x0 2019-12-11T17:31:25.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.370+0000 7f7282026ec0 10 moving default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL to cache LRU end 2019-12-11T17:31:25.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.370+0000 7f7282026ec0 20 get_system_obj_state: rctx=0x7ffd7c33d8a8 obj=default.rgw.meta:users.keys:KYHHQNSRBQRKUNJKBGDL state=0x556da6622940 s->prefetch_data=0 2019-12-11T17:31:25.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.370+0000 7f7282026ec0 10 cache get: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL : type miss (requested=0x6, cached=0x0) 2019-12-11T17:31:25.379 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.375+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL info.flags=0x0 2019-12-11T17:31:25.379 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.375+0000 7f7282026ec0 10 moving default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL to cache LRU end 2019-12-11T17:31:25.444 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.440+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T17:31:25.444 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.440+0000 7f7282026ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T17:31:25.444 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.440+0000 7f7282026ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=636 2019-12-11T17:31:25.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.449+0000 7f7254ff9700 10 RGWWatcher::handle_notify() notify_id 592705486879 cookie 93929430612912 notifier 4613 bl.length()=636 2019-12-11T17:31:25.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.449+0000 7f7254ff9700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T17:31:25.453 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.449+0000 7f7254ff9700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T17:31:25.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.671+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-11T17:31:25.675 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.671+0000 7f7282026ec0 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-11T17:31:25.676 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.671+0000 7f7282026ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=194 2019-12-11T17:31:25.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.681+0000 7f7254ff9700 10 RGWWatcher::handle_notify() notify_id 592705486898 cookie 93929430641008 notifier 4613 bl.length()=194 2019-12-11T17:31:25.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.681+0000 7f7254ff9700 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-11T17:31:25.685 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.681+0000 7f7254ff9700 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-11T17:31:25.844 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.840+0000 7f7282026ec0 10 cache put: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL info.flags=0x7 2019-12-11T17:31:25.844 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.840+0000 7f7282026ec0 10 moving default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL to cache LRU end 2019-12-11T17:31:25.845 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.840+0000 7f7282026ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=186 2019-12-11T17:31:25.856 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.852+0000 7f7254ff9700 10 RGWWatcher::handle_notify() notify_id 592705486920 cookie 93929430618128 notifier 4613 bl.length()=186 2019-12-11T17:31:25.856 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.852+0000 7f7254ff9700 10 cache put: name=default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL info.flags=0x7 2019-12-11T17:31:25.857 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:25.852+0000 7f7254ff9700 10 moving default.rgw.meta+users.keys+KYHHQNSRBQRKUNJKBGDL to cache LRU end 2019-12-11T17:31:26.664 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T17:31:26.664 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "foo.client.0", 2019-12-11T17:31:26.664 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Mr. foo.client.0", 2019-12-11T17:31:26.664 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "foo.client.0+test@test.test", 2019-12-11T17:31:26.665 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T17:31:26.665 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T17:31:26.665 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [], 2019-12-11T17:31:26.665 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [ 2019-12-11T17:31:26.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T17:31:26.665 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "foo.client.0", 2019-12-11T17:31:26.665 INFO:teuthology.orchestra.run.smithi079.stdout: "access_key": "KYHHQNSRBQRKUNJKBGDL", 2019-12-11T17:31:26.666 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "O/NJcEIJg1J4PeeQlnizdn2LBCPOVJHYcAWZGpS5U9dFB3lUr3zriA==" 2019-12-11T17:31:26.666 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T17:31:26.666 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T17:31:26.666 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [], 2019-12-11T17:31:26.666 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [ 2019-12-11T17:31:26.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T17:31:26.666 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "user-policy", 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: "perm": "*" 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T17:31:26.667 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T17:31:26.668 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T17:31:26.669 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T17:31:26.669 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T17:31:26.669 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T17:31:26.669 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T17:31:26.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T17:31:26.669 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T17:31:26.669 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T17:31:26.670 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T17:31:26.670 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T17:31:26.670 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:26.741 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:26.738+0000 7f7282026ec0 20 remove_watcher() i=0 2019-12-11T17:31:26.742 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:26.738+0000 7f7282026ec0 2 removed watcher, disabling cache 2019-12-11T17:31:26.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:26.769+0000 7f7282026ec0 20 remove_watcher() i=1 2019-12-11T17:31:26.804 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:26.801+0000 7f7282026ec0 20 remove_watcher() i=2 2019-12-11T17:31:26.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:26.878+0000 7f7282026ec0 20 remove_watcher() i=3 2019-12-11T17:31:26.925 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:26.921+0000 7f7282026ec0 20 remove_watcher() i=4 2019-12-11T17:31:26.973 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:26.969+0000 7f7282026ec0 20 remove_watcher() i=5 2019-12-11T17:31:27.028 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.024+0000 7f7282026ec0 20 remove_watcher() i=6 2019-12-11T17:31:27.063 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.059+0000 7f7282026ec0 20 remove_watcher() i=7 2019-12-11T17:31:27.073 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 6225965774 --totp-seed OJARAYTXNKH7YNHPUDTKLRWKZ4GSP65XZCCN33C5T6LYKT2FYP4S36Z6QEP2AQXD --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2019-12-11T17:31:27.233 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.225+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.248+0000 7f23eb4a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:27.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.248+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.261 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.258+0000 7f23eb4a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:27.262 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.258+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.266 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.262+0000 7f23eb4a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:27.267 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.262+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.275 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.271+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:27.441 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.437+0000 7f23eb4a7ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:31:27.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.437+0000 7f23eb4a7ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:31:27.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.437+0000 7f23eb4a7ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:31:27.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.437+0000 7f23eb4a7ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:31:27.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.437+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.445+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:27.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.445+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.463 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.459+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:27.463 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.459+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.471 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.467+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:27.471 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.467+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.486 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.482+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:27.486 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.482+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.486+0000 7f23eb4a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:27.490 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.486+0000 7f23eb4a7ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:31:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.486+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.490+0000 7f23eb4a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.490+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.502 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.498+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:27.502 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.498+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.513 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.509+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:27.513 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.509+0000 7f23eb4a7ec0 20 zonegroup default 2019-12-11T17:31:27.513 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.509+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.518 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.514+0000 7f23eb4a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:27.518 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.514+0000 7f23eb4a7ec0 10 Cannot find current period zone using local zone 2019-12-11T17:31:27.519 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.514+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.527 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.523+0000 7f23eb4a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:27.527 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.523+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.535 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.532+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:27.536 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.532+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:27.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.545+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:27.550 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.545+0000 7f23eb4a7ec0 20 zone default found 2019-12-11T17:31:27.550 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.545+0000 7f23eb4a7ec0 20 started sync module instance, tier type = 2019-12-11T17:31:27.550 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.545+0000 7f23eb4a7ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:31:27.994 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.990+0000 7f23eb4a7ec0 20 add_watcher() i=0 2019-12-11T17:31:27.994 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.990+0000 7f23eb4a7ec0 20 add_watcher() i=1 2019-12-11T17:31:27.994 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.990+0000 7f23eb4a7ec0 20 add_watcher() i=2 2019-12-11T17:31:27.995 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.990+0000 7f23eb4a7ec0 20 add_watcher() i=3 2019-12-11T17:31:27.995 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.990+0000 7f23eb4a7ec0 20 add_watcher() i=4 2019-12-11T17:31:27.995 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.990+0000 7f23eb4a7ec0 20 add_watcher() i=5 2019-12-11T17:31:27.995 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:27.990+0000 7f23eb4a7ec0 20 add_watcher() i=6 2019-12-11T17:31:28.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:28.021+0000 7f23eb4a7ec0 20 add_watcher() i=7 2019-12-11T17:31:28.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:28.021+0000 7f23eb4a7ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:31:28.025 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:28.021+0000 7f23a8fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:31:28.046 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:28.042+0000 7f23d86d1700 0 -- >> v2:172.21.15.79:3300/0 conn(0x55b089ff42d0 msgr2=0x55b089ff66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:28.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:28.160+0000 7f23d8ed2700 0 -- 172.21.15.79:0/2788660961 >> [v2:172.21.15.79:6818/10237,v1:172.21.15.79:6819/10237] conn(0x55b08a008180 msgr2=0x55b08a00a5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:29.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.523+0000 7f231affd700 20 reqs_thread_entry: start 2019-12-11T17:31:29.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.523+0000 7f2318ff9700 20 reqs_thread_entry: start 2019-12-11T17:31:29.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.523+0000 7f23eb4a7ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:31:29.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.524+0000 7f230effd700 20 reqs_thread_entry: start 2019-12-11T17:31:29.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.524+0000 7f23eb4a7ec0 20 get_system_obj_state: rctx=0x7ffffded37e8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55b08a06ef70 s->prefetch_data=0 2019-12-11T17:31:29.529 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.524+0000 7f23eb4a7ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-11T17:31:29.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.545+0000 7f23eb4a7ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2019-12-11T17:31:29.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.545+0000 7f23eb4a7ec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T17:31:29.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.545+0000 7f23eb4a7ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T17:31:29.550 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.545+0000 7f23eb4a7ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T17:31:29.550 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.545+0000 7f23eb4a7ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T17:31:29.550 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.545+0000 7f23eb4a7ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.567+0000 7f23eb4a7ec0 20 rados_obj.operate() r=0 bl.length=451 2019-12-11T17:31:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.567+0000 7f23eb4a7ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2019-12-11T17:31:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.567+0000 7f23eb4a7ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T17:31:29.776 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.772+0000 7f23eb4a7ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T17:31:29.776 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.772+0000 7f23eb4a7ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T17:31:29.776 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.772+0000 7f23eb4a7ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=650 2019-12-11T17:31:29.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.799+0000 7f23cdffb700 10 RGWWatcher::handle_notify() notify_id 592705486880 cookie 94216717679872 notifier 4965 bl.length()=650 2019-12-11T17:31:29.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.799+0000 7f23cdffb700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T17:31:29.804 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:29.799+0000 7f23cdffb700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T17:31:30.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.237+0000 7f23eb4a7ec0 20 remove_watcher() i=0 2019-12-11T17:31:30.241 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.237+0000 7f23eb4a7ec0 2 removed watcher, disabling cache 2019-12-11T17:31:30.292 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.288+0000 7f23eb4a7ec0 20 remove_watcher() i=1 2019-12-11T17:31:30.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.331+0000 7f23eb4a7ec0 20 remove_watcher() i=2 2019-12-11T17:31:30.405 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.401+0000 7f23eb4a7ec0 20 remove_watcher() i=3 2019-12-11T17:31:30.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.454+0000 7f23eb4a7ec0 20 remove_watcher() i=4 2019-12-11T17:31:30.538 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.533+0000 7f23eb4a7ec0 20 remove_watcher() i=5 2019-12-11T17:31:30.598 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.594+0000 7f23eb4a7ec0 20 remove_watcher() i=6 2019-12-11T17:31:30.658 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.654+0000 7f23eb4a7ec0 20 remove_watcher() i=7 2019-12-11T17:31:30.669 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2019-12-11T17:31:30.670 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key ALWCWIUDSFBQNDXWYIYS --secret 1DKh92Ev1b0Fbb+i4O/ZLRlKNFchO/u4FTh40kt+eG+44ztwHQRpTQ== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2019-12-11T17:31:30.816 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.812+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:30.839 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.835+0000 7f3fcf1ecec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:30.839 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.835+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:30.847 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.843+0000 7f3fcf1ecec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:30.847 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.843+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:30.851 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.847+0000 7f3fcf1ecec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:30.851 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.847+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:30.859 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:30.855+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:31.071 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.067+0000 7f3fcf1ecec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:31:31.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.067+0000 7f3fcf1ecec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:31:31.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.067+0000 7f3fcf1ecec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:31:31.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.067+0000 7f3fcf1ecec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:31:31.072 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.067+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.078 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.074+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:31.078 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.074+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.085+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:31.089 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.085+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.097 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.093+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:31.097 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.093+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.114 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.110+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:31.115 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.110+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.116+0000 7f3fcf1ecec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:31.120 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.116+0000 7f3fcf1ecec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:31:31.121 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.116+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.122+0000 7f3fcf1ecec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:31.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.122+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.130+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:31.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.130+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.145 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.141+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:31.145 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.141+0000 7f3fcf1ecec0 20 zonegroup default 2019-12-11T17:31:31.145 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.141+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.151 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.147+0000 7f3fcf1ecec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:31.152 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.147+0000 7f3fcf1ecec0 10 Cannot find current period zone using local zone 2019-12-11T17:31:31.152 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.147+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.153+0000 7f3fcf1ecec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:31.157 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.153+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.161+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:31.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.161+0000 7f3fcf1ecec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:31.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.175+0000 7f3fcf1ecec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:31.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.176+0000 7f3fcf1ecec0 20 zone default found 2019-12-11T17:31:31.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.176+0000 7f3fcf1ecec0 20 started sync module instance, tier type = 2019-12-11T17:31:31.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.176+0000 7f3fcf1ecec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:31:31.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.567+0000 7f3fcf1ecec0 20 add_watcher() i=0 2019-12-11T17:31:31.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.567+0000 7f3fcf1ecec0 20 add_watcher() i=1 2019-12-11T17:31:31.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.567+0000 7f3fcf1ecec0 20 add_watcher() i=2 2019-12-11T17:31:31.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.567+0000 7f3fcf1ecec0 20 add_watcher() i=3 2019-12-11T17:31:31.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.567+0000 7f3fcf1ecec0 20 add_watcher() i=4 2019-12-11T17:31:31.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.567+0000 7f3fcf1ecec0 20 add_watcher() i=5 2019-12-11T17:31:31.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.567+0000 7f3fcf1ecec0 20 add_watcher() i=6 2019-12-11T17:31:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.598+0000 7f3fcf1ecec0 20 add_watcher() i=7 2019-12-11T17:31:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.598+0000 7f3fcf1ecec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:31:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.598+0000 7f3f8cfe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:31:31.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.600+0000 7f3fbbc15700 0 -- 172.21.15.79:0/296118187 >> v2:172.21.15.79:3300/0 conn(0x55eda6df4bb0 msgr2=0x55eda6df70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:31:31.643 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:31.639+0000 7f3fbc416700 0 -- 172.21.15.79:0/3258741138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eda6df8100 msgr2=0x55eda6dfa520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:32.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.961+0000 7f3efeffd700 20 reqs_thread_entry: start 2019-12-11T17:31:32.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.961+0000 7f3efcff9700 20 reqs_thread_entry: start 2019-12-11T17:31:32.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.961+0000 7f3fcf1ecec0 20 init_complete bucket index max shards: 0 2019-12-11T17:31:32.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.962+0000 7f3ef2ffd700 20 reqs_thread_entry: start 2019-12-11T17:31:32.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.962+0000 7f3fcf1ecec0 20 get_system_obj_state: rctx=0x7fffc64c4c48 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55eda6e6be20 s->prefetch_data=0 2019-12-11T17:31:32.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.962+0000 7f3fcf1ecec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2019-12-11T17:31:32.973 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.967+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2019-12-11T17:31:32.973 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.967+0000 7f3fcf1ecec0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T17:31:32.973 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.967+0000 7f3fcf1ecec0 20 get_system_obj_state: rctx=0x7fffc64c4c58 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x55eda6e6dbd0 s->prefetch_data=0 2019-12-11T17:31:32.973 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.967+0000 7f3fcf1ecec0 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2019-12-11T17:31:32.977 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.972+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2019-12-11T17:31:32.977 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.972+0000 7f3fcf1ecec0 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2019-12-11T17:31:32.977 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.972+0000 7f3fcf1ecec0 20 get_system_obj_state: rctx=0x7fffc64c4c58 obj=default.rgw.meta:users.keys:ALWCWIUDSFBQNDXWYIYS state=0x55eda6e6dbd0 s->prefetch_data=0 2019-12-11T17:31:32.977 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.972+0000 7f3fcf1ecec0 10 cache get: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS : miss 2019-12-11T17:31:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.976+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS info.flags=0x0 2019-12-11T17:31:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.976+0000 7f3fcf1ecec0 10 adding default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS to cache LRU end 2019-12-11T17:31:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.976+0000 7f3fcf1ecec0 20 get_system_obj_state: rctx=0x7fffc64c4868 obj=default.rgw.meta:users.keys:ALWCWIUDSFBQNDXWYIYS state=0x55eda6e6dbd0 s->prefetch_data=0 2019-12-11T17:31:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.976+0000 7f3fcf1ecec0 10 cache get: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS : type miss (requested=0x6, cached=0x0) 2019-12-11T17:31:32.985 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.981+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS info.flags=0x0 2019-12-11T17:31:32.985 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.981+0000 7f3fcf1ecec0 10 moving default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS to cache LRU end 2019-12-11T17:31:32.985 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.981+0000 7f3fcf1ecec0 20 get_system_obj_state: rctx=0x7fffc64c49e8 obj=default.rgw.meta:users.keys:ALWCWIUDSFBQNDXWYIYS state=0x55eda6e6dbd0 s->prefetch_data=0 2019-12-11T17:31:32.985 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.981+0000 7f3fcf1ecec0 10 cache get: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS : type miss (requested=0x6, cached=0x0) 2019-12-11T17:31:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.985+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS info.flags=0x0 2019-12-11T17:31:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:32.985+0000 7f3fcf1ecec0 10 moving default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS to cache LRU end 2019-12-11T17:31:33.037 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.033+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-11T17:31:33.037 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.033+0000 7f3fcf1ecec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T17:31:33.037 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.033+0000 7f3fcf1ecec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=710 2019-12-11T17:31:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.049+0000 7f3fb1ffb700 10 RGWWatcher::handle_notify() notify_id 592705486921 cookie 94479195166096 notifier 4977 bl.length()=710 2019-12-11T17:31:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.049+0000 7f3fb1ffb700 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-11T17:31:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.049+0000 7f3fb1ffb700 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T17:31:33.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.120+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2019-12-11T17:31:33.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.120+0000 7f3fcf1ecec0 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2019-12-11T17:31:33.124 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.120+0000 7f3fcf1ecec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=224 2019-12-11T17:31:33.143 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.139+0000 7f3fb1ffb700 10 RGWWatcher::handle_notify() notify_id 592705486899 cookie 94479195182320 notifier 4977 bl.length()=224 2019-12-11T17:31:33.143 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.139+0000 7f3fb1ffb700 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2019-12-11T17:31:33.143 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.139+0000 7f3fb1ffb700 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2019-12-11T17:31:33.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.837+0000 7f3fcf1ecec0 10 cache put: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS info.flags=0x7 2019-12-11T17:31:33.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.837+0000 7f3fcf1ecec0 10 moving default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS to cache LRU end 2019-12-11T17:31:33.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.837+0000 7f3fcf1ecec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=201 2019-12-11T17:31:33.864 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.860+0000 7f3fb1ffb700 10 RGWWatcher::handle_notify() notify_id 592705486922 cookie 94479195166096 notifier 4977 bl.length()=201 2019-12-11T17:31:33.864 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.860+0000 7f3fb1ffb700 10 cache put: name=default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS info.flags=0x7 2019-12-11T17:31:33.864 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:33.860+0000 7f3fb1ffb700 10 moving default.rgw.meta+users.keys+ALWCWIUDSFBQNDXWYIYS to cache LRU end 2019-12-11T17:31:33.970 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T17:31:33.970 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "testx$tenanteduser.client.0", 2019-12-11T17:31:33.970 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2019-12-11T17:31:33.970 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2019-12-11T17:31:33.971 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T17:31:33.971 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T17:31:33.971 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [], 2019-12-11T17:31:33.971 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [ 2019-12-11T17:31:33.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T17:31:33.971 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "testx$tenanteduser.client.0", 2019-12-11T17:31:33.971 INFO:teuthology.orchestra.run.smithi079.stdout: "access_key": "ALWCWIUDSFBQNDXWYIYS", 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "1DKh92Ev1b0Fbb+i4O/ZLRlKNFchO/u4FTh40kt+eG+44ztwHQRpTQ==" 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [], 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [ 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "user-policy", 2019-12-11T17:31:33.972 INFO:teuthology.orchestra.run.smithi079.stdout: "perm": "*" 2019-12-11T17:31:33.973 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T17:31:33.973 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T17:31:33.973 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T17:31:33.973 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T17:31:33.973 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T17:31:33.973 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T17:31:33.973 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T17:31:33.974 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T17:31:33.974 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T17:31:33.974 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T17:31:33.974 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T17:31:33.974 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T17:31:33.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T17:31:33.974 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T17:31:33.975 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T17:31:33.976 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T17:31:33.976 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T17:31:33.976 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:34.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.057+0000 7f3fcf1ecec0 20 remove_watcher() i=0 2019-12-11T17:31:34.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.057+0000 7f3fcf1ecec0 2 removed watcher, disabling cache 2019-12-11T17:31:34.109 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.105+0000 7f3fcf1ecec0 20 remove_watcher() i=1 2019-12-11T17:31:34.149 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.145+0000 7f3fcf1ecec0 20 remove_watcher() i=2 2019-12-11T17:31:34.223 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.219+0000 7f3fcf1ecec0 20 remove_watcher() i=3 2019-12-11T17:31:34.274 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.270+0000 7f3fcf1ecec0 20 remove_watcher() i=4 2019-12-11T17:31:34.322 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.318+0000 7f3fcf1ecec0 20 remove_watcher() i=5 2019-12-11T17:31:34.368 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.364+0000 7f3fcf1ecec0 20 remove_watcher() i=6 2019-12-11T17:31:34.415 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.410+0000 7f3fcf1ecec0 20 remove_watcher() i=7 2019-12-11T17:31:34.423 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 5953814340 --totp-seed 56TUG4PLBABNNFGEEF6JFDAI4X55PRG3L2YXCUDCXP6TWAR7AZGECL5CENZ7DC4G --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2019-12-11T17:31:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.575+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.612 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.603+0000 7f6daca11ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:34.612 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.603+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.615 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.611+0000 7f6daca11ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:34.615 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.611+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.620 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.616+0000 7f6daca11ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:34.620 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.616+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.628 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.624+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.820+0000 7f6daca11ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T17:31:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.820+0000 7f6daca11ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T17:31:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.820+0000 7f6daca11ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T17:31:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.820+0000 7f6daca11ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T17:31:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.820+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.829+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.829+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.848+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:34.852 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.848+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.859 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.856+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:34.860 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.856+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.877 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.874+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:34.878 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.874+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.878+0000 7f6daca11ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:34.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.878+0000 7f6daca11ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T17:31:34.882 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.878+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.886 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.882+0000 7f6daca11ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:34.886 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.882+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.890+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:34.894 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.890+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.922 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.918+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T17:31:34.922 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.918+0000 7f6daca11ec0 20 zonegroup default 2019-12-11T17:31:34.922 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.918+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.923+0000 7f6daca11ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:34.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.923+0000 7f6daca11ec0 10 Cannot find current period zone using local zone 2019-12-11T17:31:34.927 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.923+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.931 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.927+0000 7f6daca11ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T17:31:34.931 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.927+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.938 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.935+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T17:31:34.939 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.935+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:34.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.953+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T17:31:34.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.953+0000 7f6daca11ec0 20 zone default found 2019-12-11T17:31:34.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.953+0000 7f6daca11ec0 20 started sync module instance, tier type = 2019-12-11T17:31:34.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:34.953+0000 7f6daca11ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T17:31:35.333 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.329+0000 7f6daca11ec0 20 add_watcher() i=0 2019-12-11T17:31:35.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.329+0000 7f6daca11ec0 20 add_watcher() i=1 2019-12-11T17:31:35.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.329+0000 7f6daca11ec0 20 add_watcher() i=2 2019-12-11T17:31:35.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.329+0000 7f6daca11ec0 20 add_watcher() i=3 2019-12-11T17:31:35.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.329+0000 7f6daca11ec0 20 add_watcher() i=4 2019-12-11T17:31:35.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.329+0000 7f6daca11ec0 20 add_watcher() i=5 2019-12-11T17:31:35.334 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.329+0000 7f6daca11ec0 20 add_watcher() i=6 2019-12-11T17:31:35.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.370+0000 7f6daca11ec0 20 add_watcher() i=7 2019-12-11T17:31:35.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.370+0000 7f6daca11ec0 2 all 8 watchers are set, enabling cache 2019-12-11T17:31:35.374 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.370+0000 7f6d6a7dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T17:31:35.375 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.371+0000 7f6d9943a700 0 -- >> v1:172.21.15.199:6789/0 conn(0x5572a63a9db0 legacy=0x5572a63ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T17:31:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:35.459+0000 7f6d9943a700 0 -- 172.21.15.79:0/1308118106 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x5572a63bae80 msgr2=0x5572a63bd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T17:31:37.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.188+0000 7f6ce0ff9700 20 reqs_thread_entry: start 2019-12-11T17:31:37.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.188+0000 7f6cda7fc700 20 reqs_thread_entry: start 2019-12-11T17:31:37.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.188+0000 7f6daca11ec0 20 init_complete bucket index max shards: 0 2019-12-11T17:31:37.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.189+0000 7f6cd8ff9700 20 reqs_thread_entry: start 2019-12-11T17:31:37.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.189+0000 7f6daca11ec0 20 get_system_obj_state: rctx=0x7ffe89616778 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x5572a6427940 s->prefetch_data=0 2019-12-11T17:31:37.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.189+0000 7f6daca11ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2019-12-11T17:31:37.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.200+0000 7f6daca11ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2019-12-11T17:31:37.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.200+0000 7f6daca11ec0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T17:31:37.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.200+0000 7f6daca11ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T17:31:37.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.200+0000 7f6daca11ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T17:31:37.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.200+0000 7f6daca11ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T17:31:37.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.200+0000 7f6daca11ec0 20 rados->read ofs=0 len=0 2019-12-11T17:31:37.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.213+0000 7f6daca11ec0 20 rados_obj.operate() r=0 bl.length=510 2019-12-11T17:31:37.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.213+0000 7f6daca11ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2019-12-11T17:31:37.217 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.213+0000 7f6daca11ec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T17:31:37.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.372+0000 7f6daca11ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-11T17:31:37.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.372+0000 7f6daca11ec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T17:31:37.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.372+0000 7f6daca11ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=724 2019-12-11T17:31:37.387 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.383+0000 7f6d7affd700 10 RGWWatcher::handle_notify() notify_id 592705486923 cookie 93950903394848 notifier 4643 bl.length()=724 2019-12-11T17:31:37.387 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.383+0000 7f6d7affd700 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-11T17:31:37.387 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.383+0000 7f6d7affd700 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T17:31:37.946 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.942+0000 7f6daca11ec0 20 remove_watcher() i=0 2019-12-11T17:31:37.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:37.942+0000 7f6daca11ec0 2 removed watcher, disabling cache 2019-12-11T17:31:38.009 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:38.005+0000 7f6daca11ec0 20 remove_watcher() i=1 2019-12-11T17:31:38.059 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:38.055+0000 7f6daca11ec0 20 remove_watcher() i=2 2019-12-11T17:31:38.134 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:38.130+0000 7f6daca11ec0 20 remove_watcher() i=3 2019-12-11T17:31:38.152 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:59:25.082 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.163 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:59:25.093 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:38.168+0000 7f6daca11ec0 20 remove_watcher() i=4 2019-12-11T17:31:38.180 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:59:25.110 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.201 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:59:25.130 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:38.208+0000 7f6daca11ec0 20 remove_watcher() i=5 2019-12-11T17:31:38.215 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:59:25.116 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.217 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:59:25.118 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.246 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:00:59:25.176 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.250 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:38.246+0000 7f6daca11ec0 20 remove_watcher() i=6 2019-12-11T17:31:38.256 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:00:59:25.184 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.279 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:00:59:25.179 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:31:38.287 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T17:31:38.283+0000 7f6daca11ec0 20 remove_watcher() i=7 2019-12-11T17:31:38.296 INFO:tasks.s3tests:Configuring s3-tests... 2019-12-11T17:31:38.296 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2019-12-11T17:31:43.883 INFO:teuthology.orchestra.run.smithi079.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2019-12-11T17:31:43.994 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-11T17:31:44.031 INFO:teuthology.orchestra.run.smithi079.stdout:Loading mirror speeds from cached hostfile 2019-12-11T17:31:44.031 INFO:teuthology.orchestra.run.smithi079.stdout: * base: mirror.facebook.net 2019-12-11T17:31:44.032 INFO:teuthology.orchestra.run.smithi079.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-11T17:31:44.032 INFO:teuthology.orchestra.run.smithi079.stdout: * extras: mirror.facebook.net 2019-12-11T17:31:44.032 INFO:teuthology.orchestra.run.smithi079.stdout: * updates: centos.mirror.constant.com 2019-12-11T17:31:44.682 INFO:teuthology.orchestra.run.smithi079.stdout:7 packages excluded due to repository priority protections 2019-12-11T17:31:45.033 INFO:teuthology.orchestra.run.smithi079.stdout:Resolving Dependencies 2019-12-11T17:31:45.034 INFO:teuthology.orchestra.run.smithi079.stdout:--> Running transaction check 2019-12-11T17:31:45.034 INFO:teuthology.orchestra.run.smithi079.stdout:---> Package libffi-devel.x86_64 0:3.0.13-18.el7 will be installed 2019-12-11T17:31:45.423 INFO:teuthology.orchestra.run.smithi079.stdout:--> Finished Dependency Resolution 2019-12-11T17:31:45.660 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:45.661 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies Resolved 2019-12-11T17:31:45.661 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:45.661 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T17:31:45.661 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2019-12-11T17:31:45.661 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T17:31:45.661 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2019-12-11T17:31:45.661 INFO:teuthology.orchestra.run.smithi079.stdout: libffi-devel x86_64 3.0.13-18.el7 base 23 k 2019-12-11T17:31:45.662 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:45.662 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2019-12-11T17:31:45.662 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2019-12-11T17:31:45.662 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2019-12-11T17:31:45.662 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:45.662 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 23 k 2019-12-11T17:31:45.662 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 27 k 2019-12-11T17:31:45.663 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading packages: 2019-12-11T17:31:45.788 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2019-12-11T17:31:45.793 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2019-12-11T17:31:45.795 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded 2019-12-11T17:31:45.796 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2019-12-11T17:31:46.380 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libffi-devel-3.0.13-18.el7.x86_64 1/1 2019-12-11T17:31:46.783 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libffi-devel-3.0.13-18.el7.x86_64 1/1 2019-12-11T17:31:46.783 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:46.784 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2019-12-11T17:31:46.784 INFO:teuthology.orchestra.run.smithi079.stdout: libffi-devel.x86_64 0:3.0.13-18.el7 2019-12-11T17:31:46.784 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:46.785 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2019-12-11T17:31:46.956 INFO:teuthology.orchestra.run.smithi079.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2019-12-11T17:31:48.221 INFO:teuthology.orchestra.run.smithi079.stdout:Installing setuptools, pip, wheel...done. 2019-12-11T17:31:48.486 INFO:teuthology.orchestra.run.smithi079.stderr:Cache entry deserialization failed, entry ignored 2019-12-11T17:31:48.608 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting pip 2019-12-11T17:31:48.684 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl 2019-12-11T17:31:48.748 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: pip 2019-12-11T17:31:48.748 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: pip 9.0.1 2019-12-11T17:31:48.815 INFO:teuthology.orchestra.run.smithi079.stdout: Uninstalling pip-9.0.1: 2019-12-11T17:31:48.859 INFO:teuthology.orchestra.run.smithi079.stdout: Successfully uninstalled pip-9.0.1 2019-12-11T17:31:49.434 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed pip-19.3.1 2019-12-11T17:31:49.700 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T17:31:50.186 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting setuptools==32.3.1 2019-12-11T17:31:50.189 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/69/19/b1dff551058ce79d88b1e3688f1c735590d7ddf44d10681512133b35019f/setuptools-32.3.1-py2.py3-none-any.whl 2019-12-11T17:31:50.230 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: setuptools 2019-12-11T17:31:50.230 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: setuptools 28.8.0 2019-12-11T17:31:50.245 INFO:teuthology.orchestra.run.smithi079.stdout: Uninstalling setuptools-28.8.0: 2019-12-11T17:31:50.254 INFO:teuthology.orchestra.run.smithi079.stdout: Successfully uninstalled setuptools-28.8.0 2019-12-11T17:31:50.380 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed setuptools-32.3.1 2019-12-11T17:31:50.808 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T17:31:51.031 INFO:teuthology.orchestra.run.smithi079.stdout:Processing /home/ubuntu/.cache/pip/wheels/54/b7/c7/2ada654ee54483c9329871665aaf4a6056c3ce36f29cf66e67/PyYAML-5.2-cp27-cp27mu-linux_x86_64.whl 2019-12-11T17:31:51.092 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting nose>=1.0.0 2019-12-11T17:31:51.094 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl 2019-12-11T17:31:51.199 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting boto>=2.6.0 2019-12-11T17:31:51.205 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl 2019-12-11T17:31:51.732 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting boto3>=1.0.0 2019-12-11T17:31:51.786 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/be/04/7744980154abb220792503968cf0aac28a19f5c9be974635c747b553adf1/boto3-1.10.36-py2.py3-none-any.whl (128kB) 2019-12-11T17:31:51.894 INFO:teuthology.orchestra.run.smithi079.stdout:Processing /home/ubuntu/.cache/pip/wheels/56/0f/19/fbbf81e5764e6d8b74501c4357a88c14c94466ec777c03734c/bunch-1.0.1-cp27-none-any.whl 2019-12-11T17:31:52.257 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting gevent>=1.0 2019-12-11T17:31:52.273 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/e6/e9/3a693414f706e12abe60554cd73c5ae8f848b182ae58018f93d86c9eb418/gevent-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-11T17:31:52.612 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting isodate>=0.4.4 2019-12-11T17:31:52.622 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB) 2019-12-11T17:31:52.806 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting requests>=0.14.0 2019-12-11T17:31:52.816 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB) 2019-12-11T17:31:53.049 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting pytz>=2011k 2019-12-11T17:31:53.058 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB) 2019-12-11T17:31:53.257 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting ordereddict 2019-12-11T17:31:53.267 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/53/25/ef88e8e45db141faa9598fbf7ad0062df8f50f881a36ed6a0073e1572126/ordereddict-1.1.tar.gz 2019-12-11T17:31:53.541 INFO:teuthology.orchestra.run.smithi079.stdout:Processing /home/ubuntu/.cache/pip/wheels/51/ab/65/f318b4271019823cf7d7c178ac76a8ea4d6e6271fe9f5ba488/httplib2-0.14.0-cp27-none-any.whl 2019-12-11T17:31:53.926 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting lxml 2019-12-11T17:31:53.940 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/25/73/730ad7249847f741c7e622f52b971daa5540f6fb87589bc92c717f5aafba/lxml-4.4.2-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-11T17:31:54.109 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting jmespath<1.0.0,>=0.7.1 2019-12-11T17:31:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl 2019-12-11T17:31:54.725 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting botocore<1.14.0,>=1.13.36 2019-12-11T17:31:54.735 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/08/e0/e47a793322fd68381e4cda4f98740c558c9ef2a87e71d06589048f59cbcb/botocore-1.13.36-py2.py3-none-any.whl (5.8MB) 2019-12-11T17:31:55.375 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting s3transfer<0.3.0,>=0.2.0 2019-12-11T17:31:55.387 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl (70kB) 2019-12-11T17:31:55.588 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" 2019-12-11T17:31:55.589 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/8b/6e/f2d25875713ad0885c8d3c69269697406652e6f64e1a6bd8264f7a609327/greenlet-0.4.15-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-11T17:31:55.654 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting six 2019-12-11T17:31:55.664 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl 2019-12-11T17:31:55.740 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting chardet<3.1.0,>=3.0.2 2019-12-11T17:31:55.749 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB) 2019-12-11T17:31:55.841 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting idna<2.9,>=2.5 2019-12-11T17:31:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB) 2019-12-11T17:31:55.993 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 2019-12-11T17:31:56.005 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl (125kB) 2019-12-11T17:31:56.133 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting certifi>=2017.4.17 2019-12-11T17:31:56.144 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl (156kB) 2019-12-11T17:31:56.245 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting docutils<0.16,>=0.10 2019-12-11T17:31:56.256 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/3a/dc/bf2b15d1fa15a6f7a9e77a61b74ecbbae7258558fcda8ffc9a6638a6b327/docutils-0.15.2-py2-none-any.whl (548kB) 2019-12-11T17:31:56.417 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting python-dateutil<2.8.1,>=2.1; python_version >= "2.7" 2019-12-11T17:31:56.428 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB) 2019-12-11T17:31:56.531 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" 2019-12-11T17:31:56.541 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/d8/a6/f46ae3f1da0cd4361c344888f59ec2f5785e69c872e175a748ef6071cdb5/futures-3.3.0-py2-none-any.whl 2019-12-11T17:31:56.573 INFO:teuthology.orchestra.run.smithi079.stdout:Building wheels for collected packages: ordereddict 2019-12-11T17:31:56.573 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for ordereddict (setup.py): started 2019-12-11T17:31:56.790 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for ordereddict (setup.py): finished with status 'done' 2019-12-11T17:31:56.790 INFO:teuthology.orchestra.run.smithi079.stdout: Created wheel for ordereddict: filename=ordereddict-1.1-cp27-none-any.whl size=3327 sha256=5017518ea512f56218ea5823583660356aa68027601412d9cd14baca0ff421f0 2019-12-11T17:31:56.790 INFO:teuthology.orchestra.run.smithi079.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/78/d3/a8/9f52c7a389a0ffba6f575a3886b5f7a55461fb6ec34aa5fd38 2019-12-11T17:31:56.792 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully built ordereddict 2019-12-11T17:31:56.874 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: PyYAML, nose, boto, jmespath, docutils, six, python-dateutil, urllib3, botocore, futures, s3transfer, boto3, bunch, greenlet, gevent, isodate, chardet, idna, certifi, requests, pytz, ordereddict, httplib2, lxml 2019-12-11T17:31:59.030 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed PyYAML-5.2 boto-2.49.0 boto3-1.10.36 botocore-1.13.36 bunch-1.0.1 certifi-2019.11.28 chardet-3.0.4 docutils-0.15.2 futures-3.3.0 gevent-1.4.0 greenlet-0.4.15 httplib2-0.14.0 idna-2.8 isodate-0.6.0 jmespath-0.9.4 lxml-4.4.2 nose-1.3.7 ordereddict-1.1 python-dateutil-2.8.0 pytz-2019.3 requests-2.22.0 s3transfer-0.2.1 six-1.13.0 urllib3-1.25.7 2019-12-11T17:31:59.396 INFO:teuthology.orchestra.run.smithi079.stdout:running develop 2019-12-11T17:31:59.461 INFO:teuthology.orchestra.run.smithi079.stdout:running egg_info 2019-12-11T17:31:59.462 INFO:teuthology.orchestra.run.smithi079.stdout:creating s3tests.egg-info 2019-12-11T17:31:59.462 INFO:teuthology.orchestra.run.smithi079.stdout:writing requirements to s3tests.egg-info/requires.txt 2019-12-11T17:31:59.462 INFO:teuthology.orchestra.run.smithi079.stdout:writing s3tests.egg-info/PKG-INFO 2019-12-11T17:31:59.462 INFO:teuthology.orchestra.run.smithi079.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2019-12-11T17:31:59.462 INFO:teuthology.orchestra.run.smithi079.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2019-12-11T17:31:59.463 INFO:teuthology.orchestra.run.smithi079.stdout:writing entry points to s3tests.egg-info/entry_points.txt 2019-12-11T17:31:59.463 INFO:teuthology.orchestra.run.smithi079.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2019-12-11T17:31:59.465 INFO:teuthology.orchestra.run.smithi079.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2019-12-11T17:31:59.466 INFO:teuthology.orchestra.run.smithi079.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2019-12-11T17:31:59.467 INFO:teuthology.orchestra.run.smithi079.stdout:running build_ext 2019-12-11T17:31:59.468 INFO:teuthology.orchestra.run.smithi079.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/s3tests.egg-link (link to .) 2019-12-11T17:31:59.468 INFO:teuthology.orchestra.run.smithi079.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2019-12-11T17:31:59.471 INFO:teuthology.orchestra.run.smithi079.stdout:Installing s3tests-fuzz-headers script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-11T17:31:59.471 INFO:teuthology.orchestra.run.smithi079.stdout:Installing s3tests-generate-objects script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-11T17:31:59.471 INFO:teuthology.orchestra.run.smithi079.stdout:Installing s3tests-test-roundtrip script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-11T17:31:59.471 INFO:teuthology.orchestra.run.smithi079.stdout:Installing s3tests-analysis-rwstats script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-11T17:31:59.472 INFO:teuthology.orchestra.run.smithi079.stdout:Installing s3tests-test-readwrite script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-11T17:31:59.472 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.472 INFO:teuthology.orchestra.run.smithi079.stdout:Installed /home/ubuntu/cephtest/s3-tests 2019-12-11T17:31:59.472 INFO:teuthology.orchestra.run.smithi079.stdout:Processing dependencies for s3tests==0.0.1 2019-12-11T17:31:59.521 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for isodate==0.6.0 2019-12-11T17:31:59.521 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: isodate 0.6.0 2019-12-11T17:31:59.521 INFO:teuthology.orchestra.run.smithi079.stdout:Adding isodate 0.6.0 to easy-install.pth file 2019-12-11T17:31:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.522 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.523 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for gevent==1.4.0 2019-12-11T17:31:59.523 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: gevent 1.4.0 2019-12-11T17:31:59.523 INFO:teuthology.orchestra.run.smithi079.stdout:Adding gevent 1.4.0 to easy-install.pth file 2019-12-11T17:31:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.524 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.524 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for bunch==1.0.1 2019-12-11T17:31:59.524 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: bunch 1.0.1 2019-12-11T17:31:59.524 INFO:teuthology.orchestra.run.smithi079.stdout:Adding bunch 1.0.1 to easy-install.pth file 2019-12-11T17:31:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.524 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.526 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for PyYAML==5.2 2019-12-11T17:31:59.526 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: PyYAML 5.2 2019-12-11T17:31:59.526 INFO:teuthology.orchestra.run.smithi079.stdout:Adding PyYAML 5.2 to easy-install.pth file 2019-12-11T17:31:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.526 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.526 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for boto3==1.10.36 2019-12-11T17:31:59.527 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: boto3 1.10.36 2019-12-11T17:31:59.527 INFO:teuthology.orchestra.run.smithi079.stdout:Adding boto3 1.10.36 to easy-install.pth file 2019-12-11T17:31:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.527 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.527 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for boto==2.49.0 2019-12-11T17:31:59.527 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: boto 2.49.0 2019-12-11T17:31:59.527 INFO:teuthology.orchestra.run.smithi079.stdout:Adding boto 2.49.0 to easy-install.pth file 2019-12-11T17:31:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.528 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.529 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for six==1.13.0 2019-12-11T17:31:59.529 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: six 1.13.0 2019-12-11T17:31:59.529 INFO:teuthology.orchestra.run.smithi079.stdout:Adding six 1.13.0 to easy-install.pth file 2019-12-11T17:31:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.530 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.530 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for greenlet==0.4.15 2019-12-11T17:31:59.530 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: greenlet 0.4.15 2019-12-11T17:31:59.530 INFO:teuthology.orchestra.run.smithi079.stdout:Adding greenlet 0.4.15 to easy-install.pth file 2019-12-11T17:31:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.531 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.531 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for s3transfer==0.2.1 2019-12-11T17:31:59.531 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: s3transfer 0.2.1 2019-12-11T17:31:59.534 INFO:teuthology.orchestra.run.smithi079.stdout:Adding s3transfer 0.2.1 to easy-install.pth file 2019-12-11T17:31:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.534 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.534 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for botocore==1.13.36 2019-12-11T17:31:59.534 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: botocore 1.13.36 2019-12-11T17:31:59.534 INFO:teuthology.orchestra.run.smithi079.stdout:Adding botocore 1.13.36 to easy-install.pth file 2019-12-11T17:31:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.535 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.535 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for jmespath==0.9.4 2019-12-11T17:31:59.535 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: jmespath 0.9.4 2019-12-11T17:31:59.535 INFO:teuthology.orchestra.run.smithi079.stdout:Adding jmespath 0.9.4 to easy-install.pth file 2019-12-11T17:31:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.535 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.536 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for futures==3.3.0 2019-12-11T17:31:59.536 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: futures 3.3.0 2019-12-11T17:31:59.536 INFO:teuthology.orchestra.run.smithi079.stdout:Adding futures 3.3.0 to easy-install.pth file 2019-12-11T17:31:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.536 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.536 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for urllib3==1.25.7 2019-12-11T17:31:59.536 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: urllib3 1.25.7 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout:Adding urllib3 1.25.7 to easy-install.pth file 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for python-dateutil==2.8.0 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: python-dateutil 2.8.0 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout:Adding python-dateutil 2.8.0 to easy-install.pth file 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.537 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.538 INFO:teuthology.orchestra.run.smithi079.stdout:Searching for docutils==0.15.2 2019-12-11T17:31:59.538 INFO:teuthology.orchestra.run.smithi079.stdout:Best match: docutils 0.15.2 2019-12-11T17:31:59.538 INFO:teuthology.orchestra.run.smithi079.stdout:Adding docutils 0.15.2 to easy-install.pth file 2019-12-11T17:31:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T17:31:59.538 INFO:teuthology.orchestra.run.smithi079.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-11T17:31:59.538 INFO:teuthology.orchestra.run.smithi079.stdout:Finished processing dependencies for s3tests==0.0.1 2019-12-11T17:31:59.545 INFO:teuthology.orchestra.run.smithi079:> cat > /home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2019-12-11T17:31:59.592 INFO:tasks.s3tests:Configuring boto... 2019-12-11T17:31:59.593 INFO:teuthology.orchestra.run.smithi079:> cat > /home/ubuntu/cephtest/boto.cfg 2019-12-11T17:31:59.681 INFO:teuthology.orchestra.run.smithi079:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!fails_with_subdomain' 2019-12-11T17:32:04.360 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2019-12-11T17:32:06.726 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2019-12-11T17:32:10.849 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2019-12-11T17:32:13.795 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2019-12-11T17:32:17.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2019-12-11T17:32:19.171 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2019-12-11T17:32:24.344 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2019-12-11T17:32:26.612 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2019-12-11T17:32:27.284 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2019-12-11T17:32:27.459 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2019-12-11T17:32:28.871 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2019-12-11T17:32:31.381 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2019-12-11T17:32:33.373 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2019-12-11T17:32:35.234 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2019-12-11T17:32:35.326 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2019-12-11T17:32:35.397 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2019-12-11T17:32:35.397 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2019-12-11T17:32:35.397 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2019-12-11T17:32:35.397 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2019-12-11T17:32:35.397 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2019-12-11T17:32:35.397 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2019-12-11T17:32:35.398 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2019-12-11T17:32:35.398 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2019-12-11T17:32:35.398 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2019-12-11T17:32:35.398 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2019-12-11T17:32:35.398 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2019-12-11T17:32:35.400 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2019-12-11T17:32:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2019-12-11T17:32:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2019-12-11T17:32:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2019-12-11T17:32:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2019-12-11T17:32:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2019-12-11T17:32:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2019-12-11T17:32:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2019-12-11T17:32:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2019-12-11T17:32:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2019-12-11T17:32:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2019-12-11T17:32:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2019-12-11T17:32:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2019-12-11T17:32:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2019-12-11T17:32:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2019-12-11T17:32:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2019-12-11T17:32:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2019-12-11T17:32:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2019-12-11T17:32:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2019-12-11T17:32:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2019-12-11T17:32:41.753 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2019-12-11T17:32:41.753 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_lifecycle_transition ... SKIP 2019-12-11T17:32:41.754 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2019-12-11T17:32:41.754 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2019-12-11T17:32:41.754 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2019-12-11T17:32:41.754 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2019-12-11T17:32:41.754 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2019-12-11T17:32:41.754 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2019-12-11T17:32:41.755 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2019-12-11T17:32:41.755 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2019-12-11T17:32:47.494 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2019-12-11T17:32:54.929 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2019-12-11T17:33:03.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2019-12-11T17:33:07.262 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2019-12-11T17:33:11.998 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2019-12-11T17:33:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_enc ... ok 2019-12-11T17:33:18.540 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2019-12-11T17:33:24.046 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2019-12-11T17:33:24.047 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2019-12-11T17:33:24.047 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2019-12-11T17:33:24.047 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2019-12-11T17:33:24.047 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2019-12-11T17:33:24.047 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2019-12-11T17:33:24.047 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2019-12-11T17:33:24.048 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2019-12-11T17:33:24.048 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2019-12-11T17:33:24.048 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2019-12-11T17:33:24.048 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2019-12-11T17:33:24.048 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2019-12-11T17:33:24.048 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2019-12-11T17:33:24.048 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2019-12-11T17:33:24.049 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2019-12-11T17:33:24.049 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2019-12-11T17:33:24.049 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2019-12-11T17:33:24.049 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2019-12-11T17:33:24.049 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2019-12-11T17:33:24.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2019-12-11T17:33:24.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2019-12-11T17:33:24.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2019-12-11T17:33:24.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2019-12-11T17:33:24.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2019-12-11T17:33:24.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2019-12-11T17:33:24.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2019-12-11T17:33:24.052 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2019-12-11T17:33:24.052 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2019-12-11T17:33:24.052 INFO:teuthology.orchestra.run.smithi079.stderr:SKIP 2019-12-11T17:33:24.055 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.functional.test_utils.test_generate ... ok 2019-12-11T17:34:49.162 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_load_graph ... ok 2019-12-11T17:34:49.162 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_leaf_node ... ok 2019-12-11T17:34:49.162 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_node ... ok 2019-12-11T17:34:49.163 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_bad_node ... ok 2019-12-11T17:34:49.163 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_nonexistant_child ... ok 2019-12-11T17:34:49.163 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_printable ... ok 2019-12-11T17:34:49.163 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_binary ... ok 2019-12-11T17:34:49.507 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_printable_no_whitespace ... ok 2019-12-11T17:34:49.663 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_binary_no_whitespace ... ok 2019-12-11T17:34:50.008 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_no_args ... ok 2019-12-11T17:34:50.031 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_no_charset ... ok 2019-12-11T17:34:50.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_exact_length ... ok 2019-12-11T17:34:50.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_bad_charset ... ok 2019-12-11T17:34:50.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_missing_length ... ok 2019-12-11T17:34:50.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_assemble_decision ... ok 2019-12-11T17:34:50.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_escape ... ok 2019-12-11T17:34:50.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_indirect ... ok 2019-12-11T17:34:50.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_indirect_double ... ok 2019-12-11T17:34:50.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive ... ok 2019-12-11T17:34:50.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_mutual ... ok 2019-12-11T17:34:50.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_not_too_eager ... ok 2019-12-11T17:34:50.051 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_make_choice_unweighted_with_space ... ok 2019-12-11T17:34:50.055 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_choices ... ok 2019-12-11T17:34:50.055 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_null_choices ... ok 2019-12-11T17:34:50.055 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_null_choices ... ok 2019-12-11T17:34:50.055 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_null_child ... ok 2019-12-11T17:34:50.060 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_set ... ok 2019-12-11T17:34:50.060 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_header_presence ... ok 2019-12-11T17:34:50.060 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_duplicate_header ... ok 2019-12-11T17:34:50.060 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_headers ... ok 2019-12-11T17:34:50.086 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFileValidator.test_new_file_is_valid ... ok 2019-12-11T17:34:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_on_several_calls ... ok 2019-12-11T17:34:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_when_size_is_1 ... ok 2019-12-11T17:34:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFiles.test_random_file_valid ... ok 2019-12-11T17:34:52.899 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2019-12-11T17:35:02.684 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2019-12-11T17:35:04.193 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2019-12-11T17:35:08.319 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2019-12-11T17:35:13.375 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2019-12-11T17:35:17.098 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2019-12-11T17:35:21.154 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2019-12-11T17:35:23.170 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2019-12-11T17:35:24.574 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:03:11.504 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.579 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:03:11.509 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.591 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:03:11.520 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.592 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:03:11.493 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.595 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:03:11.525 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.598 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:03:11.499 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.749 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:03:11.649 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.756 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:03:11.687 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:24.774 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:03:11.704 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:35:26.469 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2019-12-11T17:35:28.008 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2019-12-11T17:35:31.420 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2019-12-11T17:35:32.788 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2019-12-11T17:35:34.961 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2019-12-11T17:35:36.945 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2019-12-11T17:35:37.104 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2019-12-11T17:35:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2019-12-11T17:35:44.788 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2019-12-11T17:35:49.913 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2019-12-11T17:35:52.425 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2019-12-11T17:35:55.460 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2019-12-11T17:35:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2019-12-11T17:36:01.605 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2019-12-11T17:36:03.800 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2019-12-11T17:36:06.029 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2019-12-11T17:36:08.546 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2019-12-11T17:36:08.621 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2019-12-11T17:36:08.684 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2019-12-11T17:36:08.778 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2019-12-11T17:36:08.893 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2019-12-11T17:36:09.138 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2019-12-11T17:36:12.771 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2019-12-11T17:36:18.237 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2019-12-11T17:36:26.929 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2019-12-11T17:36:31.813 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2019-12-11T17:36:40.322 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2019-12-11T17:36:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2019-12-11T17:36:54.415 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2019-12-11T17:37:02.260 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2019-12-11T17:37:09.706 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2019-12-11T17:37:20.978 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2019-12-11T17:37:32.407 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2019-12-11T17:37:35.650 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2019-12-11T17:37:39.382 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2019-12-11T17:37:45.741 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2019-12-11T17:37:50.624 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2019-12-11T17:38:00.918 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2019-12-11T17:38:14.381 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2019-12-11T17:38:20.495 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2019-12-11T17:38:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2019-12-11T17:38:35.696 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2019-12-11T17:38:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2019-12-11T17:38:50.413 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2019-12-11T17:38:57.134 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2019-12-11T17:39:04.038 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2019-12-11T17:39:10.043 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2019-12-11T17:39:17.412 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2019-12-11T17:39:24.381 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2019-12-11T17:39:31.792 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2019-12-11T17:39:39.249 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2019-12-11T17:39:43.899 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2019-12-11T17:39:49.428 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2019-12-11T17:39:55.679 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2019-12-11T17:40:02.786 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2019-12-11T17:40:04.705 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:07:51.635 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.727 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:07:51.657 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.730 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:07:51.660 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.732 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:07:51.663 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.768 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:07:51.670 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.772 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:07:51.673 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.871 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:07:51.801 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.889 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:07:51.790 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:04.920 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:07:51.849 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:06.826 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2019-12-11T17:40:35.948 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:08:22.879 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:35.954 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:08:22.884 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:35.963 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:08:22.893 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:35.966 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:08:22.897 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:35.967 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:08:22.869 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:35.974 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:08:22.876 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:36.004 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:08:22.934 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:36.012 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:08:22.942 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:40:36.027 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:08:22.928 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:49.961 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:13:36.892 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:49.973 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:13:36.903 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:49.977 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:13:36.908 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:49.981 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:13:36.911 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:49.982 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:13:36.883 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:49.988 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:13:36.890 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:50.156 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:13:37.086 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:50.169 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:13:37.070 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:45:50.170 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:13:37.100 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.161 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:16:13.092 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.190 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:16:13.121 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.223 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:16:13.124 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.225 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:16:13.126 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.238 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:16:13.168 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.240 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:16:13.170 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.264 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:16:13.195 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.304 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:16:13.205 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:48:26.318 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:16:13.248 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.741 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:18:48.672 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.753 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:18:48.684 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.761 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:18:48.691 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.766 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:18:48.667 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.767 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:18:48.698 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.775 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:18:48.676 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.866 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:18:48.796 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.881 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:18:48.812 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:51:01.898 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:18:48.799 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.544 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:24:00.474 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.564 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:24:00.484 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.564 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:24:00.494 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.568 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:24:00.498 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.571 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:24:00.473 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.579 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:24:00.481 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.779 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:24:00.678 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.800 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:24:00.730 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:13.811 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:24:00.741 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.594 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:24:34.524 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.630 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:24:34.560 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.637 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:24:34.567 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.641 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:24:34.569 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.665 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:24:34.595 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.669 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:24:34.599 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.682 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:24:34.583 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.689 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:24:34.590 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:56:47.728 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:24:34.629 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T17:58:39.459 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2019-12-11T17:58:45.742 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2019-12-11T17:58:51.324 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2019-12-11T17:58:56.341 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2019-12-11T17:59:00.769 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2019-12-11T17:59:07.330 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2019-12-11T17:59:11.546 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2019-12-11T17:59:16.286 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2019-12-11T17:59:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2019-12-11T17:59:26.921 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2019-12-11T17:59:33.433 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2019-12-11T17:59:40.493 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2019-12-11T17:59:44.123 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2019-12-11T17:59:50.342 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2019-12-11T17:59:57.680 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2019-12-11T18:00:02.614 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2019-12-11T18:00:10.575 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2019-12-11T18:00:16.025 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2019-12-11T18:00:24.911 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2019-12-11T18:00:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2019-12-11T18:00:35.305 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2019-12-11T18:00:42.759 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2019-12-11T18:00:47.886 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2019-12-11T18:00:57.080 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2019-12-11T18:01:02.813 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2019-12-11T18:01:10.735 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2019-12-11T18:01:18.886 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2019-12-11T18:01:24.498 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:29:11.428 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.502 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:29:11.432 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.518 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:29:11.448 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.527 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:29:11.454 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.536 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:29:11.438 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.549 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:29:11.451 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.720 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:29:11.650 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.721 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:29:11.651 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:24.794 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:29:11.695 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:01:26.447 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2019-12-11T18:01:32.505 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2019-12-11T18:02:09.444 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2019-12-11T18:02:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2019-12-11T18:02:47.180 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2019-12-11T18:02:53.605 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2019-12-11T18:03:00.325 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2019-12-11T18:03:09.208 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2019-12-11T18:03:14.983 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2019-12-11T18:03:22.533 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2019-12-11T18:03:28.328 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2019-12-11T18:03:35.608 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2019-12-11T18:03:41.476 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2019-12-11T18:03:48.177 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2019-12-11T18:03:57.847 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2019-12-11T18:04:14.772 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2019-12-11T18:04:18.676 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2019-12-11T18:04:21.494 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2019-12-11T18:04:23.293 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2019-12-11T18:04:24.987 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2019-12-11T18:04:25.145 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2019-12-11T18:04:25.290 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2019-12-11T18:04:25.408 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2019-12-11T18:04:28.686 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2019-12-11T18:04:34.658 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2019-12-11T18:04:34.903 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2019-12-11T18:04:40.966 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2019-12-11T18:04:43.323 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_read_notexist ... ok 2019-12-11T18:04:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2019-12-11T18:04:50.950 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2019-12-11T18:04:59.028 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2019-12-11T18:05:01.352 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2019-12-11T18:05:03.543 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2019-12-11T18:05:07.125 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2019-12-11T18:05:11.219 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2019-12-11T18:05:14.121 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:33:01.051 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.144 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:33:01.074 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.162 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:33:01.064 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.164 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:33:01.093 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.181 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:33:01.110 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.199 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:33:01.101 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.299 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:33:01.229 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.306 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:33:01.235 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:14.321 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:33:01.222 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:05:16.400 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2019-12-11T18:05:18.769 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2019-12-11T18:05:20.793 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2019-12-11T18:05:25.221 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2019-12-11T18:05:28.238 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_unicode_metadata ... ok 2019-12-11T18:05:31.976 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2019-12-11T18:05:34.226 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2019-12-11T18:05:37.074 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2019-12-11T18:05:41.502 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2019-12-11T18:05:45.896 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2019-12-11T18:05:48.586 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2019-12-11T18:05:51.072 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2019-12-11T18:05:54.204 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2019-12-11T18:06:00.300 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2019-12-11T18:06:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2019-12-11T18:06:06.560 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2019-12-11T18:06:09.315 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2019-12-11T18:06:14.317 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2019-12-11T18:06:18.954 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2019-12-11T18:06:20.977 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2019-12-11T18:06:22.793 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2019-12-11T18:06:25.391 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2019-12-11T18:06:27.490 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2019-12-11T18:06:29.544 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2019-12-11T18:06:33.065 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2019-12-11T18:06:33.822 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:34:20.751 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.830 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:34:20.760 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.843 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:34:20.772 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.846 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:34:20.775 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.852 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:34:20.754 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.860 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:34:20.762 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.892 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:34:20.821 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.937 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:34:20.866 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:33.985 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:34:20.886 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:06:35.516 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2019-12-11T18:06:38.500 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2019-12-11T18:06:40.120 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2019-12-11T18:06:43.015 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2019-12-11T18:06:44.938 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2019-12-11T18:06:47.200 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2019-12-11T18:06:49.708 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2019-12-11T18:06:52.768 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2019-12-11T18:06:54.265 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2019-12-11T18:06:57.577 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2019-12-11T18:07:01.531 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2019-12-11T18:07:03.846 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2019-12-11T18:07:06.524 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2019-12-11T18:07:09.973 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2019-12-11T18:07:12.675 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2019-12-11T18:07:16.166 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2019-12-11T18:07:22.538 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2019-12-11T18:07:26.602 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2019-12-11T18:07:30.841 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2019-12-11T18:07:35.026 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2019-12-11T18:07:39.448 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2019-12-11T18:07:44.630 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2019-12-11T18:07:50.015 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2019-12-11T18:07:56.671 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2019-12-11T18:07:58.670 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2019-12-11T18:08:04.711 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2019-12-11T18:08:09.921 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2019-12-11T18:08:13.369 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2019-12-11T18:08:18.681 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2019-12-11T18:08:22.908 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2019-12-11T18:08:31.067 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2019-12-11T18:08:38.489 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2019-12-11T18:08:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2019-12-11T18:08:44.564 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2019-12-11T18:08:51.398 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2019-12-11T18:08:55.281 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2019-12-11T18:08:59.875 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2019-12-11T18:09:02.422 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2019-12-11T18:09:06.223 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2019-12-11T18:09:11.162 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2019-12-11T18:09:16.025 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2019-12-11T18:09:24.851 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2019-12-11T18:09:30.047 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2019-12-11T18:09:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2019-12-11T18:09:34.648 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2019-12-11T18:09:37.087 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2019-12-11T18:09:40.685 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2019-12-11T18:09:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2019-12-11T18:09:48.844 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2019-12-11T18:09:51.010 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2019-12-11T18:09:54.025 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2019-12-11T18:09:54.590 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2019-12-11T18:09:55.554 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2019-12-11T18:09:56.431 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2019-12-11T18:09:56.749 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2019-12-11T18:10:00.493 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2019-12-11T18:10:03.055 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2019-12-11T18:10:05.951 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2019-12-11T18:10:08.038 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2019-12-11T18:10:09.790 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2019-12-11T18:10:13.147 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2019-12-11T18:10:15.255 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2019-12-11T18:10:17.432 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2019-12-11T18:10:21.063 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2019-12-11T18:10:24.876 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2019-12-11T18:10:29.755 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2019-12-11T18:10:33.793 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2019-12-11T18:10:37.130 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2019-12-11T18:10:41.594 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2019-12-11T18:10:46.886 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2019-12-11T18:10:53.410 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2019-12-11T18:10:58.705 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2019-12-11T18:11:01.605 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2019-12-11T18:11:08.057 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2019-12-11T18:11:11.816 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2019-12-11T18:11:16.910 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2019-12-11T18:11:21.588 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2019-12-11T18:11:25.920 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2019-12-11T18:11:32.728 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2019-12-11T18:11:37.213 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2019-12-11T18:11:41.108 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2019-12-11T18:11:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2019-12-11T18:11:45.401 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:39:32.329 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.409 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:39:32.338 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.422 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:39:32.350 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.427 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:39:32.356 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.429 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:39:32.331 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.440 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:39:32.342 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.602 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:39:32.531 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.622 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:39:32.550 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:45.667 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:39:32.569 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:11:47.614 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2019-12-11T18:11:50.292 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2019-12-11T18:11:54.947 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2019-12-11T18:11:58.252 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2019-12-11T18:12:03.618 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2019-12-11T18:12:07.564 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2019-12-11T18:12:10.181 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_notexist ... ok 2019-12-11T18:12:15.060 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2019-12-11T18:12:22.775 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2019-12-11T18:12:30.938 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2019-12-11T18:12:38.639 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2019-12-11T18:12:45.619 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2019-12-11T18:12:52.732 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2019-12-11T18:12:55.717 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:40:42.645 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.754 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:40:42.682 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.775 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:40:42.703 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.806 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:40:42.735 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.824 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:40:42.753 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.840 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:40:42.741 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.850 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:40:42.751 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.863 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:40:42.791 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:12:55.993 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:40:42.894 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:00.906 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2019-12-11T18:13:08.538 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2019-12-11T18:13:16.967 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2019-12-11T18:13:25.383 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2019-12-11T18:13:35.974 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:41:22.903 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:35.997 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:41:22.899 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.009 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:41:22.938 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.010 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:41:22.938 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.024 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:41:22.952 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.045 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:41:22.947 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.057 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:41:22.985 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.066 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:41:22.994 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.088 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:41:22.990 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:13:36.261 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2019-12-11T18:13:50.980 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2019-12-11T18:14:04.669 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2019-12-11T18:14:17.664 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2019-12-11T18:14:18.021 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2019-12-11T18:14:18.321 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2019-12-11T18:14:18.395 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2019-12-11T18:14:23.617 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2019-12-11T18:14:34.239 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2019-12-11T18:14:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2019-12-11T18:14:37.155 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2019-12-11T18:14:44.417 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2019-12-11T18:15:09.685 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2019-12-11T18:15:16.644 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2019-12-11T18:15:23.697 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2019-12-11T18:15:28.643 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2019-12-11T18:15:33.338 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2019-12-11T18:15:36.871 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2019-12-11T18:15:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2019-12-11T18:15:50.886 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2019-12-11T18:15:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2019-12-11T18:16:04.506 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2019-12-11T18:16:12.976 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2019-12-11T18:16:21.603 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2019-12-11T18:16:32.882 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2019-12-11T18:16:36.656 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2019-12-11T18:16:40.943 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2019-12-11T18:16:55.150 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:44:42.079 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.155 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:44:42.084 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.159 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:44:42.088 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.162 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:44:42.090 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.176 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:44:42.077 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.199 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:44:42.100 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.289 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:44:42.218 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.291 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:44:42.220 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:16:55.396 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:44:42.296 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:17:12.069 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2019-12-11T18:17:21.219 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2019-12-11T18:18:52.996 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2019-12-11T18:18:55.643 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2019-12-11T18:19:04.521 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2019-12-11T18:19:21.837 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:19:21.799+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:19:22.621 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2019-12-11T18:19:27.006 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2019-12-11T18:19:41.207 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2019-12-11T18:20:14.160 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:20:14.151+0000 40262700 -1 handle_data failed with exit code -5 2019-12-11T18:20:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2019-12-11T18:21:17.022 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:21:17.000+0000 40262700 -1 handle_data failed with exit code -5 2019-12-11T18:21:45.708 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:21:45.572+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:21:49.053 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2019-12-11T18:21:57.380 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:49:44.309 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.387 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:49:44.316 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.395 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:49:44.324 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.405 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:49:44.307 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.407 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:49:44.336 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.413 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:49:44.315 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.522 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:49:44.450 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.546 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:49:44.475 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:21:57.605 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:49:44.507 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.528 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:49:52.457 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.539 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:49:52.469 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.544 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:49:52.445 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.545 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:49:52.474 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.547 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:49:52.448 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.550 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:49:52.480 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.563 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:49:52.492 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.577 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:49:52.479 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:05.579 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:49:52.508 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:22:35.922 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:22:35.873+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:22:52.519 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:22:52.468+0000 40262700 -1 handle_data failed with exit code -5 2019-12-11T18:22:53.994 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:22:53.876+0000 40262700 -1 handle_data failed with exit code -5 2019-12-11T18:23:16.086 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:23:16.056+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:23:25.196 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2019-12-11T18:26:22.871 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:26:22.821+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:26:56.675 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:26:56.613+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:27:17.414 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:55:04.344 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.424 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:55:04.354 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.437 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:55:04.338 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.438 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:55:04.367 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.443 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:55:04.372 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.443 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:55:04.344 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.607 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:55:04.537 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.624 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:55:04.553 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:17.651 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:55:04.552 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:27:34.344 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:27:34.300+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:28:02.133 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:28:01.957+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:29:46.547 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2019-12-11T18:30:18.880 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:58:05.809 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:18.934 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:58:05.863 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:18.935 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:58:05.864 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:18.935 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:58:05.865 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:18.954 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:58:05.855 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:18.957 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:58:05.858 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:19.022 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:01:58:05.952 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:19.052 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:01:58:05.981 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:30:19.096 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:01:58:05.997 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:31:03.375 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2019-12-11T18:31:25.492 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:31:25.468+0000 40262700 -1 handle_data failed with exit code -5 2019-12-11T18:31:39.925 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:31:39.835+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:31:55.267 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:31:55.219+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:32:15.510 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:32:15.465+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:32:27.581 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:00:14.511 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.589 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:00:14.519 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.596 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:00:14.526 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.599 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:00:14.499 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.599 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:00:14.528 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.602 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:00:14.503 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.681 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:00:14.610 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.701 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:00:14.630 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:27.717 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:00:14.618 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:32:32.145 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:32:32.120+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-11T18:32:49.246 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T18:32:49.024+0000 40262700 -1 handle_data failed with exit code -5 2019-12-11T18:32:50.599 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2019-12-11T18:33:17.714 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2019-12-11T18:33:40.619 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2019-12-11T18:34:00.253 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2019-12-11T18:34:14.621 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2019-12-11T18:34:17.417 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2019-12-11T18:34:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2019-12-11T18:34:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_encoding ... ok 2019-12-11T18:35:01.529 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2019-12-11T18:35:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2019-12-11T18:35:11.805 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2019-12-11T18:35:16.350 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2019-12-11T18:35:37.605 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2019-12-11T18:35:45.425 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2019-12-11T18:35:53.341 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2019-12-11T18:36:01.505 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2019-12-11T18:36:11.161 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2019-12-11T18:36:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2019-12-11T18:36:38.013 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2019-12-11T18:36:51.243 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2019-12-11T18:37:15.981 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2019-12-11T18:37:26.869 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2019-12-11T18:37:39.676 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:05:26.606 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.686 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:05:26.616 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.703 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:05:26.632 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.708 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:05:26.608 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.710 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:05:26.639 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.717 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:05:26.618 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.892 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:05:26.821 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.948 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:05:26.849 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:39.955 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:05:26.884 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:37:44.866 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2019-12-11T18:38:08.108 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2019-12-11T18:38:17.563 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2019-12-11T18:38:24.373 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2019-12-11T18:38:40.411 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:06:27.340 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.414 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:06:27.344 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.443 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:06:27.373 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.457 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:06:27.358 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.470 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:06:27.400 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.485 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:06:27.386 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.492 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:06:27.422 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.518 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:06:27.447 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:40.541 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:06:27.441 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:38:41.341 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2019-12-11T18:38:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2019-12-11T18:38:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2019-12-11T18:39:00.205 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2019-12-11T18:39:03.849 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2019-12-11T18:39:07.460 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2019-12-11T18:39:10.644 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2019-12-11T18:39:17.530 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2019-12-11T18:42:51.190 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:10:38.120 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.196 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:10:38.126 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.203 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:10:38.133 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.208 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:10:38.138 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.210 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:10:38.110 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.218 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:10:38.119 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.336 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:10:38.266 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.372 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:10:38.301 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:42:51.397 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:10:38.297 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:43:41.838 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2019-12-11T18:44:27.592 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2019-12-11T18:44:34.961 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2019-12-11T18:44:51.655 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2019-12-11T18:45:01.911 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2019-12-11T18:46:49.132 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2019-12-11T18:47:01.884 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:14:48.814 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:01.893 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:14:48.822 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:01.907 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:14:48.837 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:01.913 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:14:48.842 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:02.004 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:14:48.904 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:02.007 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:14:48.908 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:02.016 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:14:48.946 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:02.018 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:14:48.948 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:47:02.122 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:14:49.023 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.151 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:15:49.081 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.157 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:15:49.087 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.167 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:15:49.097 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.170 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:15:49.100 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.176 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:15:49.076 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.193 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:15:49.093 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.219 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:15:49.149 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.223 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:15:49.152 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:02.267 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:15:49.167 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:48:29.272 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2019-12-11T18:53:15.027 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:21:01.957 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.035 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:21:01.964 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.042 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:21:01.972 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.046 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:21:01.976 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.048 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:21:01.949 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.057 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:21:01.958 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.229 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:21:02.159 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.260 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:21:02.190 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:15.305 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:21:02.204 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.137 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:21:45.067 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.167 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:21:45.097 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.216 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:21:45.146 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.247 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:21:45.148 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.252 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:21:45.182 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.263 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:21:45.191 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.263 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:21:45.162 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.312 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:21:45.213 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:53:58.313 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:21:45.238 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:13.027 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2019-12-11T18:55:23.388 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:23:10.318 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.409 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:23:10.339 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.418 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:23:10.348 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.432 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:23:10.333 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.447 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:23:10.348 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.451 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:23:10.381 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.498 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:23:10.428 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.499 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:23:10.400 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:55:23.511 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:23:10.441 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:56:56.054 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2019-12-11T18:57:36.627 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2019-12-11T18:58:15.025 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2019-12-11T18:58:23.704 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:26:10.634 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.712 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:26:10.642 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.722 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:26:10.651 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.726 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:26:10.656 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.743 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:26:10.640 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.760 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:26:10.661 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.844 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:26:10.773 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.851 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:26:10.781 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:23.938 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:26:10.839 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T18:58:37.618 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2019-12-11T18:59:07.166 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2019-12-11T18:59:13.003 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2019-12-11T18:59:36.386 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2019-12-11T18:59:51.265 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2019-12-11T18:59:55.975 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_concurrent_remove ... Exception in thread Thread-51: 2019-12-11T18:59:55.975 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T18:59:55.975 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T18:59:55.975 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T18:59:55.975 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T18:59:55.975 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T18:59:55.976 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T18:59:55.976 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T18:59:55.976 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T18:59:55.976 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T18:59:55.976 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T18:59:55.976 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T18:59:55.976 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T18:59:55.977 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T18:59:56.238 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-54: 2019-12-11T18:59:56.239 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T18:59:56.239 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T18:59:56.239 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T18:59:56.239 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T18:59:56.239 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T18:59:56.239 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T18:59:56.240 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T18:59:56.240 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T18:59:56.240 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T18:59:56.240 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T18:59:56.240 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T18:59:56.240 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T18:59:56.240 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T18:59:56.264 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-52: 2019-12-11T18:59:56.264 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T18:59:56.264 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T18:59:56.264 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T18:59:56.264 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T18:59:56.264 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T18:59:56.265 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T18:59:56.265 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T18:59:56.265 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T18:59:56.265 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T18:59:56.265 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T18:59:56.266 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T18:59:56.266 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T18:59:56.266 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T18:59:56.562 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-55: 2019-12-11T18:59:56.563 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T18:59:56.563 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T18:59:56.563 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T18:59:56.563 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T18:59:56.563 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T18:59:56.563 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T18:59:56.563 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T18:59:56.564 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T18:59:56.564 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T18:59:56.564 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T18:59:56.564 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T18:59:56.564 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T18:59:56.564 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:05.705 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-64: 2019-12-11T19:00:05.705 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:05.705 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:05.705 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:05.705 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:05.706 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:05.706 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:05.706 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:05.706 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:05.706 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:05.706 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:05.706 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:05.707 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:05.707 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:06.187 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-63: 2019-12-11T19:00:06.187 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:06.187 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:06.188 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:06.188 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:06.188 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:06.188 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:06.188 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:06.188 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:06.189 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:06.189 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:06.189 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:06.189 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:06.189 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:06.476 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-65: 2019-12-11T19:00:06.476 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:06.476 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:06.477 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:06.479 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:06.480 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:06.480 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-62: 2019-12-11T19:00:06.480 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:06.480 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:06.481 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:06.482 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:15.910 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-75: 2019-12-11T19:00:15.910 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:15.910 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:15.911 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:15.911 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:15.911 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:15.911 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:15.911 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:15.911 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:15.911 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:15.912 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:15.912 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:15.912 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:15.912 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:15.940 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-71: 2019-12-11T19:00:15.940 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:15.940 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:15.940 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:15.940 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:15.941 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:15.941 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:15.941 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:15.941 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:15.941 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:15.941 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:15.941 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:15.942 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:15.942 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:15.942 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-74: 2019-12-11T19:00:15.942 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:15.943 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:15.943 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:15.943 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:15.943 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:15.943 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:15.943 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:15.944 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:15.944 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:15.944 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:15.944 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:15.944 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:15.944 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:15.966 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-73: 2019-12-11T19:00:15.966 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:15.966 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:15.966 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:15.966 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:15.966 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:15.967 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:24.947 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-85: 2019-12-11T19:00:24.947 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:24.947 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:24.947 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:24.948 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:24.948 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:24.948 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:24.948 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:24.948 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:24.948 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:24.948 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:24.949 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:24.949 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:24.949 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:25.194 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-84: 2019-12-11T19:00:25.194 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:25.194 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:25.194 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:25.195 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:25.195 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:25.195 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:25.195 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:25.195 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:25.195 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:25.195 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:25.196 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:25.196 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:25.196 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:25.366 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-81: 2019-12-11T19:00:25.366 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:25.366 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:25.366 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:25.366 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:25.367 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:25.367 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:25.367 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:25.367 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:25.367 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:25.367 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:25.367 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:25.368 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:25.368 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:34.622 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-95: 2019-12-11T19:00:34.622 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:34.622 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:34.622 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:34.622 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:34.623 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:34.623 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:34.623 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:34.623 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:34.623 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:34.623 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:34.623 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:34.624 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:34.624 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:34.660 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-94: 2019-12-11T19:00:34.660 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:34.661 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:34.661 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:34.661 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:34.661 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:34.661 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:34.661 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:34.661 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:34.662 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:34.662 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:34.662 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:34.662 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:34.662 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:34.885 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-93: 2019-12-11T19:00:34.886 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:34.886 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:34.886 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:34.886 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:34.886 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:34.886 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:34.887 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:34.887 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:34.887 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:34.887 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:34.887 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:34.887 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:34.887 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:34.914 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-92: 2019-12-11T19:00:34.914 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:34.914 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:34.914 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:34.914 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:34.915 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:34.915 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:34.915 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:34.915 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:34.915 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:34.915 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:34.916 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:34.916 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:34.916 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:ok 2019-12-11T19:00:48.342 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... Exception in thread Thread-103: 2019-12-11T19:00:48.342 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:48.342 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:48.342 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:48.343 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:48.343 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:48.343 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:48.343 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:48.344 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:48.344 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:48.344 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:48.344 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:48.344 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:48.344 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:48.704 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-101: 2019-12-11T19:00:48.704 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:48.705 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:48.705 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:48.705 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:48.705 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:48.705 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:48.705 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:48.705 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:48.706 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:48.706 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:48.706 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:48.706 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:48.706 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:49.049 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-104: 2019-12-11T19:00:49.050 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:49.050 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:49.050 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:49.050 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:49.050 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:49.050 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:49.051 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:49.051 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:49.051 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:49.051 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:49.051 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:49.051 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:49.051 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:49.052 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-107: 2019-12-11T19:00:49.052 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:49.052 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:49.052 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:49.052 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:49.053 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:49.053 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:49.053 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:49.053 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:49.053 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:49.054 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:49.054 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:49.054 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:49.054 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:49.265 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-105: 2019-12-11T19:00:49.265 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:49.265 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:49.266 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:49.266 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:49.266 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:49.266 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:49.266 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:49.266 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:49.266 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:49.267 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:49.267 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:49.267 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:49.299 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-109: 2019-12-11T19:00:49.299 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:49.299 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:49.299 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:49.299 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:49.300 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:49.300 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:49.300 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:49.300 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:49.300 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:49.301 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:49.301 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:49.301 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:49.301 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:49.302 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-106: 2019-12-11T19:00:49.302 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:49.302 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:49.302 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:49.302 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:49.302 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:49.303 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:49.303 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:49.303 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:49.303 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:49.303 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:49.303 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:49.303 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:49.304 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:00:49.317 INFO:teuthology.orchestra.run.smithi079.stderr:Exception in thread Thread-108: 2019-12-11T19:00:49.317 INFO:teuthology.orchestra.run.smithi079.stderr:Traceback (most recent call last): 2019-12-11T19:00:49.317 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-11T19:00:49.317 INFO:teuthology.orchestra.run.smithi079.stderr: self.run() 2019-12-11T19:00:49.317 INFO:teuthology.orchestra.run.smithi079.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-11T19:00:49.318 INFO:teuthology.orchestra.run.smithi079.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-11T19:00:49.318 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-11T19:00:49.318 INFO:teuthology.orchestra.run.smithi079.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-11T19:00:49.318 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-11T19:00:49.318 INFO:teuthology.orchestra.run.smithi079.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-11T19:00:49.318 INFO:teuthology.orchestra.run.smithi079.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-11T19:00:49.318 INFO:teuthology.orchestra.run.smithi079.stderr: raise error_class(parsed_response, operation_name) 2019-12-11T19:00:49.319 INFO:teuthology.orchestra.run.smithi079.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-11T19:00:49.319 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T19:01:00.610 INFO:teuthology.orchestra.run.smithi079.stderr:ok 2019-12-11T19:01:02.962 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2019-12-11T19:01:05.778 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2019-12-11T19:01:11.446 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2019-12-11T19:01:15.777 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2019-12-11T19:01:17.964 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2019-12-11T19:01:20.309 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2019-12-11T19:01:25.067 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2019-12-11T19:01:27.690 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2019-12-11T19:01:37.999 INFO:teuthology.orchestra.run.smithi079.stderr:Check for valid x-amz-expiration header after PUT ... ok 2019-12-11T19:01:45.708 INFO:teuthology.orchestra.run.smithi079.stderr:Check for valid x-amz-expiration header on HEAD request ... ok 2019-12-11T19:01:58.069 INFO:teuthology.orchestra.run.smithi079.stderr:Check for valid x-amz-expiration header on GET request ... ok 2019-12-11T19:02:08.729 INFO:teuthology.orchestra.run.smithi079.stderr:Check x-amz-expiration-header object tags interaction ... ok 2019-12-11T19:02:16.331 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2019-12-11T19:02:21.070 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2019-12-11T19:02:24.824 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2019-12-11T19:02:29.578 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2019-12-11T19:02:34.729 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2019-12-11T19:02:38.080 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2019-12-11T19:02:41.140 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2019-12-11T19:02:46.683 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2019-12-11T19:02:50.777 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2019-12-11T19:02:54.119 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2019-12-11T19:02:57.374 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2019-12-11T19:03:01.820 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2019-12-11T19:03:03.732 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2019-12-11T19:03:06.088 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2019-12-11T19:03:09.823 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2019-12-11T19:03:11.595 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2019-12-11T19:03:34.719 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:31:21.649 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.725 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:31:21.655 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.740 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:31:21.669 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.747 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:31:21.676 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.748 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:31:21.649 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.758 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:31:21.659 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.955 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:31:21.857 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.959 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:31:21.889 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:34.961 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:31:21.890 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.273 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:31:32.202 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.295 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:31:32.224 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.309 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:31:32.238 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.316 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:31:32.245 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.320 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:31:32.222 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.324 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:31:32.225 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.332 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:31:32.261 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.340 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:31:32.270 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:03:45.369 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:31:32.271 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:04:35.308 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2019-12-11T19:05:08.390 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2019-12-11T19:05:12.482 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2019-12-11T19:05:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2019-12-11T19:05:19.156 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2019-12-11T19:05:20.626 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2019-12-11T19:05:22.712 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2019-12-11T19:06:37.612 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2019-12-11T19:07:08.548 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2019-12-11T19:07:36.038 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2019-12-11T19:07:40.872 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2019-12-11T19:07:43.543 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2019-12-11T19:07:46.074 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2019-12-11T19:07:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2019-12-11T19:07:54.958 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2019-12-11T19:07:58.336 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2019-12-11T19:08:04.977 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2019-12-11T19:08:11.046 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2019-12-11T19:08:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2019-12-11T19:08:27.992 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2019-12-11T19:08:36.491 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2019-12-11T19:08:46.214 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:36:33.143 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.221 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:36:33.150 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.231 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:36:33.160 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.235 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:36:33.164 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.236 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:36:33.137 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.243 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:36:33.144 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.403 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2019-12-11T19:08:46.426 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:36:33.355 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.460 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:36:33.388 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:46.464 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:36:33.365 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:08:53.180 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2019-12-11T19:09:02.171 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2019-12-11T19:09:08.188 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2019-12-11T19:09:13.808 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2019-12-11T19:09:19.508 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2019-12-11T19:09:27.431 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2019-12-11T19:09:33.701 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2019-12-11T19:09:41.691 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2019-12-11T19:09:50.050 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2019-12-11T19:09:53.005 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2019-12-11T19:09:58.452 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2019-12-11T19:10:04.006 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2019-12-11T19:10:11.143 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2019-12-11T19:10:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2019-12-11T19:10:25.374 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2019-12-11T19:10:39.638 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2019-12-11T19:11:54.935 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2019-12-11T19:12:04.322 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2019-12-11T19:12:11.837 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:39:58.767 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:11.842 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:39:58.771 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:11.887 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:39:58.816 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:11.889 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:39:58.818 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:11.924 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:39:58.825 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:11.933 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:39:58.834 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:11.939 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:39:58.868 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:12.004 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:39:58.933 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:12.061 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:39:58.962 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:12:14.542 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2019-12-11T19:12:24.624 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2019-12-11T19:12:39.291 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2019-12-11T19:12:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2019-12-11T19:12:54.868 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2019-12-11T19:13:05.105 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2019-12-11T19:13:17.209 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2019-12-11T19:13:20.168 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2019-12-11T19:13:22.185 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2019-12-11T19:13:24.351 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2019-12-11T19:13:26.235 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2019-12-11T19:13:28.712 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2019-12-11T19:13:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2019-12-11T19:13:33.813 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2019-12-11T19:13:35.981 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2019-12-11T19:13:38.158 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2019-12-11T19:13:49.148 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2019-12-11T19:13:53.115 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2019-12-11T19:13:57.242 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:41:44.172 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.247 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:41:44.177 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.263 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:41:44.192 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.266 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:41:44.195 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.269 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:41:44.171 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.277 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:41:44.178 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.328 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:41:44.258 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.346 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:41:44.275 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:57.381 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:41:44.281 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:13:59.449 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2019-12-11T19:14:08.858 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2019-12-11T19:14:12.158 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2019-12-11T19:14:24.721 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2019-12-11T19:14:35.544 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2019-12-11T19:14:45.554 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2019-12-11T19:14:54.617 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2019-12-11T19:15:02.526 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2019-12-11T19:15:11.338 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2019-12-11T19:15:17.924 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2019-12-11T19:15:21.514 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2019-12-11T19:15:26.678 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2019-12-11T19:15:34.779 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2019-12-11T19:15:37.869 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2019-12-11T19:15:43.882 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2019-12-11T19:15:55.564 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2019-12-11T19:16:07.504 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2019-12-11T19:16:17.732 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2019-12-11T19:16:20.475 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2019-12-11T19:16:24.213 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2019-12-11T19:16:27.275 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2019-12-11T19:16:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2019-12-11T19:16:32.900 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2019-12-11T19:16:36.173 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2019-12-11T19:16:38.400 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2019-12-11T19:16:42.526 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2019-12-11T19:16:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2019-12-11T19:16:48.726 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2019-12-11T19:16:51.296 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2019-12-11T19:16:58.090 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2019-12-11T19:17:02.933 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2019-12-11T19:17:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2019-12-11T19:19:10.945 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:46:57.875 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:10.953 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:46:57.883 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:10.960 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:46:57.889 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:10.965 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:46:57.866 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:10.966 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:46:57.895 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:10.973 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:46:57.874 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:11.165 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:46:58.094 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:11.171 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:46:58.100 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:19:11.206 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:46:58.107 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.196 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:48:20.125 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.216 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:48:20.145 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.252 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:48:20.153 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.252 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:48:20.181 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.255 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:48:20.184 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.268 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:48:20.170 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.271 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:48:20.201 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.320 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:48:20.249 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:20:33.342 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:48:20.243 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.147 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:52:10.077 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.155 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:52:10.085 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.164 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:52:10.094 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.165 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:52:10.067 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.171 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:52:10.099 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.177 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:52:10.078 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.287 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:52:10.217 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.317 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:52:10.247 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:24:23.351 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:52:10.252 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.736 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:56:41.666 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.788 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:56:41.709 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.788 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:56:41.709 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.788 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:56:41.710 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.795 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:56:41.697 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.806 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:56:41.708 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.919 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:56:41.848 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.919 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:56:41.849 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:28:54.967 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:56:41.869 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.285 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:57:20.214 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.290 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:57:20.220 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.298 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:57:20.228 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.305 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:57:20.235 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.315 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:57:20.217 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.334 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:57:20.236 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.347 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:02:57:20.277 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.350 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:02:57:20.279 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:29:33.411 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:02:57:20.313 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:35.989 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:02:22.919 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.026 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:02:22.956 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.064 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:02:22.965 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.065 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:02:22.967 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.067 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:02:22.997 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.069 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:02:22.998 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.250 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:02:23.145 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.261 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:02:23.190 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:36.280 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:02:23.210 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.514 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:02:31.444 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.523 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:02:31.453 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.534 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:02:31.464 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.536 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:02:31.438 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.537 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:02:31.466 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.545 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:02:31.446 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.551 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:02:31.481 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.564 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:02:31.493 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:34:44.606 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:02:31.508 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.585 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:05:03.515 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.598 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:05:03.528 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.603 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:05:03.533 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.624 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:05:03.554 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.663 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:05:03.565 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.669 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:05:03.570 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.681 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:05:03.611 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.722 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:05:03.652 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:37:16.765 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:05:03.667 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.664 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:07:42.594 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.675 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:07:42.605 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.679 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:07:42.581 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.680 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:07:42.610 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.684 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:07:42.585 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.689 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:07:42.619 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.784 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:07:42.713 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.791 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:07:42.692 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:39:55.836 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:07:42.765 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:07.995 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:12:54.924 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:07.999 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:12:54.929 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:08.009 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:12:54.938 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:08.014 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:12:54.943 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:08.021 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:12:54.923 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:08.028 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:12:54.929 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:08.206 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:12:55.135 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:08.235 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:12:55.164 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:08.359 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:12:55.260 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:37.986 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:13:24.915 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:37.989 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:13:24.919 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:38.015 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:13:24.944 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:38.018 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:13:24.920 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:38.028 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:13:24.954 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:38.029 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:13:24.956 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:38.033 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:13:24.935 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:38.067 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:13:24.969 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:45:38.076 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:13:25.005 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:18.797 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:18:05.727 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:18.815 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:18:05.737 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:18.815 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:18:05.744 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:18.823 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:18:05.747 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:18.826 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:18:05.728 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:18.855 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:18:05.755 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:19.020 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:18:05.950 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:19.045 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:18:05.974 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:50:19.112 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:18:06.014 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.433 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:21:46.363 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.474 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:21:46.403 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.487 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:21:46.416 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.488 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:21:46.417 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.534 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:21:46.435 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.536 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:21:46.438 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.581 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:21:46.510 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.589 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:21:46.518 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:53:59.659 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:21:46.560 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.855 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:23:16.785 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.865 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:23:16.795 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.869 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:23:16.799 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.874 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:23:16.775 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.882 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:23:16.811 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.914 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:23:16.815 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.972 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:23:16.873 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.977 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:23:16.907 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T19:55:29.988 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:23:16.918 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.768 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:28:27.698 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.781 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:28:27.711 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.801 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:28:27.731 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.805 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:28:27.735 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.832 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:28:27.733 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.834 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:28:27.736 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.956 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:28:27.886 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:40.985 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:28:27.915 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:00:41.058 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:28:27.959 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.024 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:30:07.953 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.057 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:30:07.987 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.072 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:30:08.001 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.075 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:30:07.976 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.097 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:30:08.026 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.100 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:30:08.002 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.133 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:30:08.063 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.159 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:30:08.089 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:02:21.256 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:30:08.157 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.625 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:38:29.555 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.652 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:38:29.572 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.652 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:38:29.580 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.653 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:38:29.580 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.665 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:38:29.566 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.727 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:38:29.628 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.916 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:38:29.845 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:42.957 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:38:29.887 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:10:43.001 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:38:29.902 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.763 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:38:49.693 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.769 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:38:49.699 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.778 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:38:49.679 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.778 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:38:49.708 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.781 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:38:49.683 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.784 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:38:49.714 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.802 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:38:49.732 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.818 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:38:49.748 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:11:02.824 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:38:49.725 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:13.888 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:44:00.818 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:13.903 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:44:00.829 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:13.908 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:44:00.837 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:13.911 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:44:00.841 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:13.930 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:44:00.831 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:13.952 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:44:00.854 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:14.108 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:44:01.038 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:14.176 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:44:01.106 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:16:14.198 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:44:01.099 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.198 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:46:51.128 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.222 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:46:51.130 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.222 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:46:51.148 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.222 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:46:51.149 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.277 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:46:51.207 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.278 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:46:51.179 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.286 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:46:51.187 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.329 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:46:51.259 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:19:04.372 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:46:51.273 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.635 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:49:11.565 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.640 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:49:11.571 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.651 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:49:11.581 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.657 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:49:11.587 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.665 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:49:11.566 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.676 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:49:11.577 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.728 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:49:11.658 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.758 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:49:11.687 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:21:24.818 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:49:11.720 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.195 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:54:24.125 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.224 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:54:24.123 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.225 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:54:24.131 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.225 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:54:24.139 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.225 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:54:24.143 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.232 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:54:24.134 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.386 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:54:24.316 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.398 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:54:24.329 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:26:37.480 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:54:24.381 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.611 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:55:12.542 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.653 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:55:12.577 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.654 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:55:12.581 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.666 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:55:12.596 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.677 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:55:12.578 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.681 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:55:12.583 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.692 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:55:12.622 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.704 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:55:12.634 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:27:25.716 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:55:12.617 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.400 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:59:34.330 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.403 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:59:34.334 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.439 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:59:34.340 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.442 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:59:34.343 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.443 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:59:34.373 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.447 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:59:34.377 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.584 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:03:59:34.505 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.608 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:03:59:34.509 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:31:47.652 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:03:59:34.579 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:46.976 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:03:33.907 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.028 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:03:33.958 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.047 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:03:33.949 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.052 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:03:33.954 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.064 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:03:33.994 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.067 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:03:33.997 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.158 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:03:34.088 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.181 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:03:34.081 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:35:47.194 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:03:34.124 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.275 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:04:45.205 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.285 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:04:45.216 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.294 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:04:45.224 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.298 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:04:45.228 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.304 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:04:45.205 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.319 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:04:45.221 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.343 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:04:45.273 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.358 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:04:45.288 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:36:58.452 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:04:45.354 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.689 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:09:56.619 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.699 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:09:56.627 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.709 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:09:56.640 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.715 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:09:56.645 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.718 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:09:56.617 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.724 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:09:56.624 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.900 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:09:56.829 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.924 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:09:56.826 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:42:09.941 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:09:56.871 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:43:01.551 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_load_graph ... ok 2019-12-11T20:43:01.558 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_leaf_node ... ok 2019-12-11T20:43:01.558 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_node ... ok 2019-12-11T20:43:01.558 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_bad_node ... ok 2019-12-11T20:43:01.558 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_nonexistant_child ... ok 2019-12-11T20:43:01.558 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_printable ... ok 2019-12-11T20:43:01.558 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_binary ... ok 2019-12-11T20:43:01.923 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_printable_no_whitespace ... ok 2019-12-11T20:43:02.090 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_binary_no_whitespace ... ok 2019-12-11T20:43:02.437 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_no_args ... ok 2019-12-11T20:43:02.462 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_no_charset ... ok 2019-12-11T20:43:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_exact_length ... ok 2019-12-11T20:43:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_bad_charset ... ok 2019-12-11T20:43:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_missing_length ... ok 2019-12-11T20:43:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_assemble_decision ... ok 2019-12-11T20:43:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_escape ... ok 2019-12-11T20:43:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_indirect ... ok 2019-12-11T20:43:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_indirect_double ... ok 2019-12-11T20:43:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_recursive ... ok 2019-12-11T20:43:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_recursive_mutual ... ok 2019-12-11T20:43:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_recursive_not_too_eager ... ok 2019-12-11T20:43:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_make_choice_unweighted_with_space ... ok 2019-12-11T20:43:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_weighted_choices ... ok 2019-12-11T20:43:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_null_choices ... ok 2019-12-11T20:43:02.487 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_weighted_null_choices ... ok 2019-12-11T20:43:02.487 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_null_child ... ok 2019-12-11T20:43:02.491 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_weighted_set ... ok 2019-12-11T20:43:02.491 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_header_presence ... ok 2019-12-11T20:43:02.491 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_duplicate_header ... ok 2019-12-11T20:43:02.491 INFO:teuthology.orchestra.run.smithi079.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_headers ... ok 2019-12-11T20:43:02.512 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFileValidator.test_new_file_is_valid ... ok 2019-12-11T20:43:02.533 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_on_several_calls ... ok 2019-12-11T20:43:02.533 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_when_size_is_1 ... ok 2019-12-11T20:43:02.672 INFO:teuthology.orchestra.run.smithi079.stderr:test_realistic.TestFiles.test_random_file_valid ... ok 2019-12-11T20:43:02.672 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T20:43:02.673 INFO:teuthology.orchestra.run.smithi079.stderr:---------------------------------------------------------------------- 2019-12-11T20:43:02.673 INFO:teuthology.orchestra.run.smithi079.stderr:Ran 627 tests in 11462.722s 2019-12-11T20:43:02.673 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-11T20:43:02.673 INFO:teuthology.orchestra.run.smithi079.stderr:OK (SKIP=75) 2019-12-11T20:43:02.723 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2019-12-11T20:43:02.723 INFO:teuthology.orchestra.run.smithi079:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2019-12-11T20:43:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-11T20:43:02.864 INFO:tasks.s3tests:Cleaning up boto... 2019-12-11T20:43:02.864 INFO:teuthology.orchestra.run.smithi079:> rm /home/ubuntu/cephtest/boto.cfg 2019-12-11T20:43:02.908 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2019-12-11T20:43:03.021 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.018+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.041 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.038+0000 7fa37ec9aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:03.041 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.038+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.050 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.046+0000 7fa37ec9aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:03.050 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.046+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.053 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.050+0000 7fa37ec9aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:03.054 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.050+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.058+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.248+0000 7fa37ec9aec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T20:43:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.248+0000 7fa37ec9aec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T20:43:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.248+0000 7fa37ec9aec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T20:43:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.248+0000 7fa37ec9aec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T20:43:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.248+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.259 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.256+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:03.259 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.256+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.274+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:03.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.274+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.285 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.281+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:03.285 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.281+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.299 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.295+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:03.299 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.295+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.303 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.300+0000 7fa37ec9aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:03.304 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.300+0000 7fa37ec9aec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T20:43:03.304 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.300+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.309 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.305+0000 7fa37ec9aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:03.309 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.305+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.317 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.314+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:03.318 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.314+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.335 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.332+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:03.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.332+0000 7fa37ec9aec0 20 zonegroup default 2019-12-11T20:43:03.336 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.332+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.339+0000 7fa37ec9aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:03.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.339+0000 7fa37ec9aec0 10 Cannot find current period zone using local zone 2019-12-11T20:43:03.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.339+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.350 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.346+0000 7fa37ec9aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:03.350 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.346+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.361 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.357+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:03.361 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.357+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:03.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.376+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:03.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.376+0000 7fa37ec9aec0 20 zone default found 2019-12-11T20:43:03.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.376+0000 7fa37ec9aec0 20 started sync module instance, tier type = 2019-12-11T20:43:03.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.377+0000 7fa37ec9aec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T20:43:03.789 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.785+0000 7fa37ec9aec0 20 add_watcher() i=0 2019-12-11T20:43:03.789 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.785+0000 7fa37ec9aec0 20 add_watcher() i=1 2019-12-11T20:43:03.789 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.785+0000 7fa37ec9aec0 20 add_watcher() i=2 2019-12-11T20:43:03.789 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.785+0000 7fa37ec9aec0 20 add_watcher() i=3 2019-12-11T20:43:03.790 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.785+0000 7fa37ec9aec0 20 add_watcher() i=4 2019-12-11T20:43:03.790 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.785+0000 7fa37ec9aec0 20 add_watcher() i=5 2019-12-11T20:43:03.790 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.785+0000 7fa37ec9aec0 20 add_watcher() i=6 2019-12-11T20:43:03.820 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.816+0000 7fa37ec9aec0 20 add_watcher() i=7 2019-12-11T20:43:03.820 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.816+0000 7fa37ec9aec0 2 all 8 watchers are set, enabling cache 2019-12-11T20:43:03.820 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.817+0000 7fa2bffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T20:43:03.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.823+0000 7fa36c6c5700 0 -- 172.21.15.79:0/1569237709 >> v2:172.21.15.79:3300/0 conn(0x55aa09351790 msgr2=0x55aa09351ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T20:43:03.866 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:03.862+0000 7fa36c6c5700 0 -- 172.21.15.79:0/1904112773 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7fa2ac022480 msgr2=0x7fa2ac024970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T20:43:05.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.201+0000 7fa2aa7fc700 20 reqs_thread_entry: start 2019-12-11T20:43:05.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.202+0000 7fa2a3fff700 20 reqs_thread_entry: start 2019-12-11T20:43:05.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.202+0000 7fa37ec9aec0 20 init_complete bucket index max shards: 0 2019-12-11T20:43:05.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.202+0000 7fa2a27fc700 20 reqs_thread_entry: start 2019-12-11T20:43:05.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.202+0000 7fa37ec9aec0 20 get_system_obj_state: rctx=0x7ffc9f9bd5a8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55aa093d68c0 s->prefetch_data=0 2019-12-11T20:43:05.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.202+0000 7fa37ec9aec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-11T20:43:05.229 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.226+0000 7fa37ec9aec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2019-12-11T20:43:05.229 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.226+0000 7fa37ec9aec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T20:43:05.230 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.226+0000 7fa37ec9aec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T20:43:05.230 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.226+0000 7fa37ec9aec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T20:43:05.230 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.226+0000 7fa37ec9aec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T20:43:05.230 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.226+0000 7fa37ec9aec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:05.264 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.253+0000 7fa37ec9aec0 20 rados_obj.operate() r=0 bl.length=465 2019-12-11T20:43:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.253+0000 7fa37ec9aec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2019-12-11T20:43:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.253+0000 7fa37ec9aec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T20:43:05.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.273+0000 7fa37ec9aec0 10 removing key index: SYWZYEEVOCKRYXJJFPLE 2019-12-11T20:43:05.277 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.273+0000 7fa37ec9aec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=170 2019-12-11T20:43:05.294 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.291+0000 7fa3517fa700 10 RGWWatcher::handle_notify() notify_id 592705486973 cookie 94188787191600 notifier 4998 bl.length()=170 2019-12-11T20:43:05.557 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.553+0000 7fa37ec9aec0 10 removing email index: bar.client.0+test@test.test 2019-12-11T20:43:05.557 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.553+0000 7fa37ec9aec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=178 2019-12-11T20:43:05.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:05.564+0000 7fa3517fa700 10 RGWWatcher::handle_notify() notify_id 592705486911 cookie 94188787186128 notifier 4998 bl.length()=178 2019-12-11T20:43:06.266 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:06.263+0000 7fa37ec9aec0 10 removing user buckets index 2019-12-11T20:43:06.266 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:06.263+0000 7fa37ec9aec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=169 2019-12-11T20:43:06.283 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:06.279+0000 7fa3517fa700 10 RGWWatcher::handle_notify() notify_id 592705487814 cookie 94188787187984 notifier 4998 bl.length()=169 2019-12-11T20:43:06.547 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:06.544+0000 7fa37ec9aec0 10 removing user index: bar.client.0 2019-12-11T20:43:06.548 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:06.544+0000 7fa37ec9aec0 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2019-12-11T20:43:06.548 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:06.544+0000 7fa37ec9aec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=161 2019-12-11T20:43:06.564 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:06.561+0000 7fa3517fa700 10 RGWWatcher::handle_notify() notify_id 592705486912 cookie 94188787186128 notifier 4998 bl.length()=161 2019-12-11T20:43:07.155 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.152+0000 7fa37ec9aec0 20 remove_watcher() i=0 2019-12-11T20:43:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.152+0000 7fa37ec9aec0 2 removed watcher, disabling cache 2019-12-11T20:43:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.192+0000 7fa37ec9aec0 20 remove_watcher() i=1 2019-12-11T20:43:07.227 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.224+0000 7fa37ec9aec0 20 remove_watcher() i=2 2019-12-11T20:43:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.307+0000 7fa37ec9aec0 20 remove_watcher() i=3 2019-12-11T20:43:07.380 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.375+0000 7fa37ec9aec0 20 remove_watcher() i=4 2019-12-11T20:43:07.433 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.429+0000 7fa37ec9aec0 20 remove_watcher() i=5 2019-12-11T20:43:07.485 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.481+0000 7fa37ec9aec0 20 remove_watcher() i=6 2019-12-11T20:43:07.549 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.545+0000 7fa37ec9aec0 20 remove_watcher() i=7 2019-12-11T20:43:07.556 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2019-12-11T20:43:07.660 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.656+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.676+0000 7fa2fe84eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:07.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.676+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.684+0000 7fa2fe84eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:07.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.684+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.692 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.688+0000 7fa2fe84eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:07.692 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.688+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.698 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.695+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:07.889 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.885+0000 7fa2fe84eec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T20:43:07.889 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.885+0000 7fa2fe84eec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T20:43:07.889 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.885+0000 7fa2fe84eec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T20:43:07.889 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.885+0000 7fa2fe84eec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T20:43:07.889 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.885+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.897 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.894+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:07.898 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.894+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.908+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:07.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.909+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.918+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:07.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.918+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.934 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.931+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:07.934 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.931+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.938+0000 7fa2fe84eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.938+0000 7fa2fe84eec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T20:43:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.938+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.943+0000 7fa2fe84eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.943+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.954+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:07.957 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.954+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.975 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.972+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:07.975 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.972+0000 7fa2fe84eec0 20 zonegroup default 2019-12-11T20:43:07.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.972+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.978+0000 7fa2fe84eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:07.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.978+0000 7fa2fe84eec0 10 Cannot find current period zone using local zone 2019-12-11T20:43:07.982 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.978+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.986 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.983+0000 7fa2fe84eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:07.987 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.983+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:07.997 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.994+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:07.997 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:07.994+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:08.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.010+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:08.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.010+0000 7fa2fe84eec0 20 zone default found 2019-12-11T20:43:08.014 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.010+0000 7fa2fe84eec0 20 started sync module instance, tier type = 2019-12-11T20:43:08.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.010+0000 7fa2fe84eec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T20:43:08.519 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.515+0000 7fa2fe84eec0 20 add_watcher() i=0 2019-12-11T20:43:08.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.515+0000 7fa2fe84eec0 20 add_watcher() i=1 2019-12-11T20:43:08.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.515+0000 7fa2fe84eec0 20 add_watcher() i=2 2019-12-11T20:43:08.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.515+0000 7fa2fe84eec0 20 add_watcher() i=3 2019-12-11T20:43:08.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.515+0000 7fa2fe84eec0 20 add_watcher() i=4 2019-12-11T20:43:08.520 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.515+0000 7fa2fe84eec0 20 add_watcher() i=5 2019-12-11T20:43:08.521 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.515+0000 7fa2fe84eec0 20 add_watcher() i=6 2019-12-11T20:43:08.552 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.548+0000 7fa2fe84eec0 20 add_watcher() i=7 2019-12-11T20:43:08.552 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.548+0000 7fa2fe84eec0 2 all 8 watchers are set, enabling cache 2019-12-11T20:43:08.552 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.548+0000 7fa23ffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T20:43:08.557 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.553+0000 7fa2eba78700 0 -- 172.21.15.79:0/2951588317 >> v2:172.21.15.79:3300/0 conn(0x55a69238ac80 msgr2=0x55a692391560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T20:43:08.614 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:08.611+0000 7fa2eba78700 0 -- 172.21.15.79:0/2644539320 >> [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] conn(0x55a6923a2e20 msgr2=0x55a6923a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-11T20:43:10.434 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.431+0000 7fa22a7fc700 20 reqs_thread_entry: start 2019-12-11T20:43:10.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.431+0000 7fa223fff700 20 reqs_thread_entry: start 2019-12-11T20:43:10.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.431+0000 7fa2fe84eec0 20 init_complete bucket index max shards: 0 2019-12-11T20:43:10.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.432+0000 7fa2227fc700 20 reqs_thread_entry: start 2019-12-11T20:43:10.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.432+0000 7fa2fe84eec0 20 get_system_obj_state: rctx=0x7ffd3c683b18 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55a69240f830 s->prefetch_data=0 2019-12-11T20:43:10.436 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.432+0000 7fa2fe84eec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-11T20:43:10.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.455+0000 7fa2fe84eec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2019-12-11T20:43:10.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.455+0000 7fa2fe84eec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T20:43:10.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.455+0000 7fa2fe84eec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T20:43:10.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.455+0000 7fa2fe84eec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T20:43:10.459 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.455+0000 7fa2fe84eec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T20:43:10.460 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.455+0000 7fa2fe84eec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:10.486 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.483+0000 7fa2fe84eec0 20 rados_obj.operate() r=0 bl.length=465 2019-12-11T20:43:10.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.483+0000 7fa2fe84eec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2019-12-11T20:43:10.487 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.483+0000 7fa2fe84eec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T20:43:10.504 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.500+0000 7fa2fe84eec0 10 removing key index: KYHHQNSRBQRKUNJKBGDL 2019-12-11T20:43:10.504 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.501+0000 7fa2fe84eec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=170 2019-12-11T20:43:10.512 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.509+0000 7fa2e17fa700 10 RGWWatcher::handle_notify() notify_id 592705487815 cookie 94173906032448 notifier 4664 bl.length()=170 2019-12-11T20:43:10.995 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.991+0000 7fa2fe84eec0 10 removing email index: foo.client.0+test@test.test 2019-12-11T20:43:10.995 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:10.991+0000 7fa2fe84eec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=178 2019-12-11T20:43:11.015 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.011+0000 7fa2e17fa700 10 RGWWatcher::handle_notify() notify_id 592705486920 cookie 94173906054896 notifier 4664 bl.length()=178 2019-12-11T20:43:11.595 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.592+0000 7fa2fe84eec0 10 removing user buckets index 2019-12-11T20:43:11.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.592+0000 7fa2fe84eec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=169 2019-12-11T20:43:11.607 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.603+0000 7fa2e17fa700 10 RGWWatcher::handle_notify() notify_id 592705486913 cookie 94173906027536 notifier 4664 bl.length()=169 2019-12-11T20:43:11.958 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.954+0000 7fa2fe84eec0 10 removing user index: foo.client.0 2019-12-11T20:43:11.958 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.954+0000 7fa2fe84eec0 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2019-12-11T20:43:11.958 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.954+0000 7fa2fe84eec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=161 2019-12-11T20:43:11.978 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:11.974+0000 7fa2e17fa700 10 RGWWatcher::handle_notify() notify_id 592705486882 cookie 94173906025568 notifier 4664 bl.length()=161 2019-12-11T20:43:12.286 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.282+0000 7fa2fe84eec0 20 remove_watcher() i=0 2019-12-11T20:43:12.286 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.282+0000 7fa2fe84eec0 2 removed watcher, disabling cache 2019-12-11T20:43:12.326 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.322+0000 7fa2fe84eec0 20 remove_watcher() i=1 2019-12-11T20:43:12.356 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.352+0000 7fa2fe84eec0 20 remove_watcher() i=2 2019-12-11T20:43:12.403 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.395+0000 7fa2fe84eec0 20 remove_watcher() i=3 2019-12-11T20:43:12.435 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.431+0000 7fa2fe84eec0 20 remove_watcher() i=4 2019-12-11T20:43:12.468 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.464+0000 7fa2fe84eec0 20 remove_watcher() i=5 2019-12-11T20:43:12.506 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.502+0000 7fa2fe84eec0 20 remove_watcher() i=6 2019-12-11T20:43:12.537 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.534+0000 7fa2fe84eec0 20 remove_watcher() i=7 2019-12-11T20:43:12.545 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2019-12-11T20:43:12.669 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.665+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.698+0000 7f632ee09ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:12.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.698+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.711 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.708+0000 7f632ee09ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:12.712 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.708+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.718 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.715+0000 7f632ee09ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:12.718 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.715+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.727 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.723+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:12.900 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.897+0000 7f632ee09ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T20:43:12.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.897+0000 7f632ee09ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T20:43:12.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.897+0000 7f632ee09ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T20:43:12.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.897+0000 7f632ee09ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T20:43:12.901 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.897+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.908+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:12.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.908+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.929 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.926+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:12.929 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.926+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.941 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.937+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:12.942 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.937+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.954 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.951+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:12.954 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.951+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.957+0000 7f632ee09ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:12.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.957+0000 7f632ee09ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T20:43:12.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.957+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.962+0000 7f632ee09ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:12.966 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.962+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:12.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.972+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:12.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:12.972+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:13.011 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.008+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:13.012 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.008+0000 7f632ee09ec0 20 zonegroup default 2019-12-11T20:43:13.012 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.008+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:13.022 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.014+0000 7f632ee09ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:13.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.014+0000 7f632ee09ec0 10 Cannot find current period zone using local zone 2019-12-11T20:43:13.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.014+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:13.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.019+0000 7f632ee09ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:13.023 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.019+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:13.032 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.028+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:13.033 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.028+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:13.051 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.047+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:13.051 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.047+0000 7f632ee09ec0 20 zone default found 2019-12-11T20:43:13.051 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.047+0000 7f632ee09ec0 20 started sync module instance, tier type = 2019-12-11T20:43:13.051 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.047+0000 7f632ee09ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T20:43:13.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.381+0000 7f632ee09ec0 20 add_watcher() i=0 2019-12-11T20:43:13.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.381+0000 7f632ee09ec0 20 add_watcher() i=1 2019-12-11T20:43:13.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.381+0000 7f632ee09ec0 20 add_watcher() i=2 2019-12-11T20:43:13.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.381+0000 7f632ee09ec0 20 add_watcher() i=3 2019-12-11T20:43:13.386 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.381+0000 7f632ee09ec0 20 add_watcher() i=4 2019-12-11T20:43:13.386 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.381+0000 7f632ee09ec0 20 add_watcher() i=5 2019-12-11T20:43:13.386 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.381+0000 7f632ee09ec0 20 add_watcher() i=6 2019-12-11T20:43:13.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.416+0000 7f632ee09ec0 20 add_watcher() i=7 2019-12-11T20:43:13.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.416+0000 7f632ee09ec0 2 all 8 watchers are set, enabling cache 2019-12-11T20:43:13.420 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.416+0000 7f62ecfe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T20:43:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.417+0000 7f631c033700 0 -- >> v1:172.21.15.79:6789/0 conn(0x560c14d26fb0 legacy=0x560c14d293d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2019-12-11T20:43:13.451 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:13.448+0000 7f631c033700 0 -- 172.21.15.79:0/3576524931 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x560c14d2c320 msgr2=0x560c14d2e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T20:43:14.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.838+0000 7f625effd700 20 reqs_thread_entry: start 2019-12-11T20:43:14.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.838+0000 7f625cff9700 20 reqs_thread_entry: start 2019-12-11T20:43:14.842 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.838+0000 7f632ee09ec0 20 init_complete bucket index max shards: 0 2019-12-11T20:43:14.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.839+0000 7f6252ffd700 20 reqs_thread_entry: start 2019-12-11T20:43:14.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.839+0000 7f632ee09ec0 20 get_system_obj_state: rctx=0x7ffca328e3c8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x560c14da5dc0 s->prefetch_data=0 2019-12-11T20:43:14.843 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.839+0000 7f632ee09ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2019-12-11T20:43:14.854 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.851+0000 7f632ee09ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2019-12-11T20:43:14.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.851+0000 7f632ee09ec0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T20:43:14.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.851+0000 7f632ee09ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-11T20:43:14.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.851+0000 7f632ee09ec0 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2019-12-11T20:43:14.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.851+0000 7f632ee09ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-11T20:43:14.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.851+0000 7f632ee09ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-11T20:43:14.855 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.851+0000 7f632ee09ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:14.869 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.864+0000 7f632ee09ec0 20 rados_obj.operate() r=0 bl.length=524 2019-12-11T20:43:14.870 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.864+0000 7f632ee09ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2019-12-11T20:43:14.870 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.864+0000 7f632ee09ec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-11T20:43:14.879 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.876+0000 7f632ee09ec0 10 removing key index: ALWCWIUDSFBQNDXWYIYS 2019-12-11T20:43:14.879 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.876+0000 7f632ee09ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=170 2019-12-11T20:43:14.888 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:14.884+0000 7f6311ffb700 10 RGWWatcher::handle_notify() notify_id 592705487816 cookie 94609888849808 notifier 4996 bl.length()=170 2019-12-11T20:43:15.234 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.230+0000 7f632ee09ec0 10 removing email index: testx$tenanteduser.client.0+test@test.test 2019-12-11T20:43:15.234 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.230+0000 7f632ee09ec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=193 2019-12-11T20:43:15.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.248+0000 7f6311ffb700 10 RGWWatcher::handle_notify() notify_id 592705486921 cookie 94609888868224 notifier 4996 bl.length()=193 2019-12-11T20:43:15.632 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.628+0000 7f632ee09ec0 10 removing user buckets index 2019-12-11T20:43:15.632 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.628+0000 7f632ee09ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=184 2019-12-11T20:43:15.649 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.646+0000 7f6311ffb700 10 RGWWatcher::handle_notify() notify_id 592705486922 cookie 94609888854352 notifier 4996 bl.length()=184 2019-12-11T20:43:15.801 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.798+0000 7f632ee09ec0 10 removing user index: testx$tenanteduser.client.0 2019-12-11T20:43:15.801 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.798+0000 7f632ee09ec0 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2019-12-11T20:43:15.802 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.798+0000 7f632ee09ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=176 2019-12-11T20:43:15.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:15.807+0000 7f6311ffb700 10 RGWWatcher::handle_notify() notify_id 592705487817 cookie 94609888849808 notifier 4996 bl.length()=176 2019-12-11T20:43:16.342 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.339+0000 7f632ee09ec0 20 remove_watcher() i=0 2019-12-11T20:43:16.343 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.339+0000 7f632ee09ec0 2 removed watcher, disabling cache 2019-12-11T20:43:16.405 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.401+0000 7f632ee09ec0 20 remove_watcher() i=1 2019-12-11T20:43:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.458+0000 7f632ee09ec0 20 remove_watcher() i=2 2019-12-11T20:43:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.503+0000 7f632ee09ec0 20 remove_watcher() i=3 2019-12-11T20:43:16.547 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.544+0000 7f632ee09ec0 20 remove_watcher() i=4 2019-12-11T20:43:16.594 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.590+0000 7f632ee09ec0 20 remove_watcher() i=5 2019-12-11T20:43:16.656 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.649+0000 7f632ee09ec0 20 remove_watcher() i=6 2019-12-11T20:43:16.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:16.704+0000 7f632ee09ec0 20 remove_watcher() i=7 2019-12-11T20:43:16.715 INFO:tasks.s3tests:Removing s3-tests... 2019-12-11T20:43:16.716 INFO:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/s3-tests 2019-12-11T20:43:16.850 INFO:teuthology.run_tasks:Running task swift... 2019-12-11T20:43:16.863 INFO:tasks.swift:clients=['client.0'] 2019-12-11T20:43:16.863 INFO:tasks.swift:Downloading swift... 2019-12-11T20:43:16.864 INFO:teuthology.orchestra.run.smithi079:> git clone -b ceph-master git://git.ceph.com/git/swift.git /home/ubuntu/cephtest/swift 2019-12-11T20:43:17.001 INFO:teuthology.orchestra.run.smithi079.stdout:Cloning into '/home/ubuntu/cephtest/swift'... 2019-12-11T20:43:17.407 INFO:tasks.swift:Creating rgw users... 2019-12-11T20:43:17.408 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph user create --subuser foo.client.0:foo --display-name 'Mr. foo.client.0 foo' --secret PT3Io5Q7rZ1FF6aKn6gYJsC3KBhJ5X3RFRLZvQKgS0SCZIDwOcRtqw== --email foo.client.0+test@test.test --key-type swift --access full 2019-12-11T20:43:17.575 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.572+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.596 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.593+0000 7f37da262ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:17.597 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.593+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.611 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.606+0000 7f37da262ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:17.611 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.606+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.612+0000 7f37da262ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:17.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.612+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.627 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.623+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:17.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.823+0000 7f37da262ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T20:43:17.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.823+0000 7f37da262ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T20:43:17.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.823+0000 7f37da262ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T20:43:17.827 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.823+0000 7f37da262ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T20:43:17.828 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.823+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.838 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.834+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:17.838 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.835+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.861 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.858+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:17.862 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.858+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.876 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.873+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:17.876 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.873+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.889+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:17.893 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.889+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.896+0000 7f37da262ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:17.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.896+0000 7f37da262ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T20:43:17.899 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.896+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.905 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.902+0000 7f37da262ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:17.906 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.902+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.912+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:17.915 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.912+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.925 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.922+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:17.925 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.922+0000 7f37da262ec0 20 zonegroup default 2019-12-11T20:43:17.926 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.922+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.931 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.928+0000 7f37da262ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:17.931 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.928+0000 7f37da262ec0 10 Cannot find current period zone using local zone 2019-12-11T20:43:17.932 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.928+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.936 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.933+0000 7f37da262ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:17.936 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.933+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.943+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:17.947 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.943+0000 7f37da262ec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:17.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.956+0000 7f37da262ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:17.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.956+0000 7f37da262ec0 20 zone default found 2019-12-11T20:43:17.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.956+0000 7f37da262ec0 20 started sync module instance, tier type = 2019-12-11T20:43:17.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:17.956+0000 7f37da262ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T20:43:18.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.316+0000 7f37da262ec0 20 add_watcher() i=0 2019-12-11T20:43:18.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.316+0000 7f37da262ec0 20 add_watcher() i=1 2019-12-11T20:43:18.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.316+0000 7f37da262ec0 20 add_watcher() i=2 2019-12-11T20:43:18.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.316+0000 7f37da262ec0 20 add_watcher() i=3 2019-12-11T20:43:18.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.316+0000 7f37da262ec0 20 add_watcher() i=4 2019-12-11T20:43:18.320 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.316+0000 7f37da262ec0 20 add_watcher() i=5 2019-12-11T20:43:18.321 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.316+0000 7f37da262ec0 20 add_watcher() i=6 2019-12-11T20:43:18.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.351+0000 7f37da262ec0 20 add_watcher() i=7 2019-12-11T20:43:18.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.351+0000 7f37da262ec0 2 all 8 watchers are set, enabling cache 2019-12-11T20:43:18.355 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.351+0000 7f3797fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T20:43:18.362 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.359+0000 7f37c7c8d700 0 -- 172.21.15.79:0/3161624452 >> v2:172.21.15.199:3300/0 conn(0x55d654e0f7b0 msgr2=0x55d654e0fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-11T20:43:18.408 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:18.404+0000 7f37c748c700 0 -- 172.21.15.79:0/1435190785 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7f3708022470 msgr2=0x7f3708024980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T20:43:19.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.741+0000 7f37067fc700 20 reqs_thread_entry: start 2019-12-11T20:43:19.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.742+0000 7f36fffff700 20 reqs_thread_entry: start 2019-12-11T20:43:19.745 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.742+0000 7f37da262ec0 20 init_complete bucket index max shards: 0 2019-12-11T20:43:19.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.742+0000 7f36fe7fc700 20 reqs_thread_entry: start 2019-12-11T20:43:19.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.743+0000 7f37da262ec0 20 get_system_obj_state: rctx=0x7ffc50646a38 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55d654e93720 s->prefetch_data=0 2019-12-11T20:43:19.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.743+0000 7f37da262ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-11T20:43:19.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.754+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2019-12-11T20:43:19.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.754+0000 7f37da262ec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T20:43:19.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.755+0000 7f37da262ec0 20 get_system_obj_state: rctx=0x7ffc50646a48 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x55d654e954b0 s->prefetch_data=0 2019-12-11T20:43:19.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.755+0000 7f37da262ec0 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2019-12-11T20:43:19.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.764+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2019-12-11T20:43:19.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.764+0000 7f37da262ec0 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-11T20:43:19.767 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.764+0000 7f37da262ec0 20 get_system_obj_state: rctx=0x7ffc50646658 obj=default.rgw.meta:users.swift:foo.client.0:foo state=0x55d654e954b0 s->prefetch_data=0 2019-12-11T20:43:19.768 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.764+0000 7f37da262ec0 10 cache get: name=default.rgw.meta+users.swift+foo.client.0:foo : miss 2019-12-11T20:43:19.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.771+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x0 2019-12-11T20:43:19.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.771+0000 7f37da262ec0 10 adding default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-11T20:43:19.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.771+0000 7f37da262ec0 20 get_system_obj_state: rctx=0x7ffc506467d8 obj=default.rgw.meta:users.swift:foo.client.0:foo state=0x55d654e95760 s->prefetch_data=0 2019-12-11T20:43:19.775 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.771+0000 7f37da262ec0 10 cache get: name=default.rgw.meta+users.swift+foo.client.0:foo : type miss (requested=0x6, cached=0x0) 2019-12-11T20:43:19.782 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.779+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x0 2019-12-11T20:43:19.783 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.779+0000 7f37da262ec0 10 moving default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-11T20:43:19.848 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.845+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T20:43:19.849 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.845+0000 7f37da262ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T20:43:19.849 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.845+0000 7f37da262ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=612 2019-12-11T20:43:19.874 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.870+0000 7f37acff9700 10 RGWWatcher::handle_notify() notify_id 592705486883 cookie 94379035298928 notifier 4685 bl.length()=612 2019-12-11T20:43:19.874 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.870+0000 7f37acff9700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T20:43:19.874 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:19.870+0000 7f37acff9700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T20:43:20.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.373+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-11T20:43:20.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.373+0000 7f37da262ec0 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-11T20:43:20.377 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.373+0000 7f37da262ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=194 2019-12-11T20:43:20.396 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.393+0000 7f37acff9700 10 RGWWatcher::handle_notify() notify_id 592705486923 cookie 94379035340688 notifier 4685 bl.length()=194 2019-12-11T20:43:20.397 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.393+0000 7f37acff9700 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-11T20:43:20.397 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.393+0000 7f37acff9700 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-11T20:43:20.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.698+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x7 2019-12-11T20:43:20.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.698+0000 7f37da262ec0 10 moving default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-11T20:43:20.702 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.698+0000 7f37da262ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=183 2019-12-11T20:43:20.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.706+0000 7f37acff9700 10 RGWWatcher::handle_notify() notify_id 592705486914 cookie 94379035300928 notifier 4685 bl.length()=183 2019-12-11T20:43:20.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.706+0000 7f37acff9700 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x7 2019-12-11T20:43:20.710 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:20.706+0000 7f37acff9700 10 moving default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-11T20:43:21.582 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.578+0000 7f37da262ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T20:43:21.582 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.578+0000 7f37da262ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T20:43:21.582 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.578+0000 7f37da262ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=636 2019-12-11T20:43:21.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.599+0000 7f37acff9700 10 RGWWatcher::handle_notify() notify_id 592705486884 cookie 94379035298928 notifier 4685 bl.length()=636 2019-12-11T20:43:21.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.599+0000 7f37acff9700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-11T20:43:21.603 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.599+0000 7f37acff9700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-11T20:43:21.826 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T20:43:21.826 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "foo.client.0", 2019-12-11T20:43:21.826 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Mr. foo.client.0 foo", 2019-12-11T20:43:21.827 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "foo.client.0+test@test.test", 2019-12-11T20:43:21.827 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T20:43:21.827 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T20:43:21.827 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [ 2019-12-11T20:43:21.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T20:43:21.827 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "foo.client.0:foo", 2019-12-11T20:43:21.827 INFO:teuthology.orchestra.run.smithi079.stdout: "permissions": "full-control" 2019-12-11T20:43:21.828 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T20:43:21.828 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T20:43:21.828 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [], 2019-12-11T20:43:21.828 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [ 2019-12-11T20:43:21.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T20:43:21.828 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "foo.client.0:foo", 2019-12-11T20:43:21.829 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "PT3Io5Q7rZ1FF6aKn6gYJsC3KBhJ5X3RFRLZvQKgS0SCZIDwOcRtqw==" 2019-12-11T20:43:21.829 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T20:43:21.829 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T20:43:21.829 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [], 2019-12-11T20:43:21.829 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T20:43:21.829 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T20:43:21.829 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T20:43:21.830 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T20:43:21.830 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T20:43:21.830 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T20:43:21.830 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T20:43:21.830 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T20:43:21.830 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T20:43:21.831 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T20:43:21.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T20:43:21.831 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T20:43:21.831 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T20:43:21.831 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T20:43:21.831 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T20:43:21.831 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T20:43:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T20:43:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T20:43:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T20:43:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T20:43:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T20:43:21.832 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T20:43:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T20:43:21.887 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.883+0000 7f37da262ec0 20 remove_watcher() i=0 2019-12-11T20:43:21.887 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.883+0000 7f37da262ec0 2 removed watcher, disabling cache 2019-12-11T20:43:21.934 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.931+0000 7f37da262ec0 20 remove_watcher() i=1 2019-12-11T20:43:21.976 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:21.970+0000 7f37da262ec0 20 remove_watcher() i=2 2019-12-11T20:43:22.045 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.041+0000 7f37da262ec0 20 remove_watcher() i=3 2019-12-11T20:43:22.110 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.106+0000 7f37da262ec0 20 remove_watcher() i=4 2019-12-11T20:43:22.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.166+0000 7f37da262ec0 20 remove_watcher() i=5 2019-12-11T20:43:22.231 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.227+0000 7f37da262ec0 20 remove_watcher() i=6 2019-12-11T20:43:22.282 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.278+0000 7f37da262ec0 20 remove_watcher() i=7 2019-12-11T20:43:22.292 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph user create --subuser bar.client.0:bar --display-name 'Mr. bar.client.0 bar' --secret DO0GrnskfuSWBt+yWdfYDDC9yZ3tzBfvzUhxGh1HDr7ZDzcBEg2TdQ== --email bar.client.0+test@test.test --key-type swift --access full 2019-12-11T20:43:22.470 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.466+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.496+0000 7f58d466fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:22.500 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.496+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.506+0000 7f58d466fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:22.510 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.506+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.514 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.511+0000 7f58d466fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:22.515 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.511+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.522 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.519+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:22.705 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.702+0000 7f58d466fec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-11T20:43:22.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.702+0000 7f58d466fec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-11T20:43:22.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.702+0000 7f58d466fec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-11T20:43:22.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.702+0000 7f58d466fec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-11T20:43:22.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.702+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.716 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.712+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:22.716 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.712+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.730 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.727+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:22.731 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.727+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.740 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.736+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:22.740 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.736+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.751 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.748+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:22.752 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.748+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.754+0000 7f58d466fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:22.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.754+0000 7f58d466fec0 10 cannot find current period zonegroup using local zonegroup 2019-12-11T20:43:22.758 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.754+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.760+0000 7f58d466fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:22.764 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.760+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.769+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:22.773 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.769+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.794 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.787+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-11T20:43:22.794 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.787+0000 7f58d466fec0 20 zonegroup default 2019-12-11T20:43:22.794 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.787+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.798 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.794+0000 7f58d466fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:22.798 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.794+0000 7f58d466fec0 10 Cannot find current period zone using local zone 2019-12-11T20:43:22.798 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.794+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.799+0000 7f58d466fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-11T20:43:22.803 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.799+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.808+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-11T20:43:22.811 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.808+0000 7f58d466fec0 20 rados->read ofs=0 len=0 2019-12-11T20:43:22.831 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.827+0000 7f58d466fec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-11T20:43:22.831 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.827+0000 7f58d466fec0 20 zone default found 2019-12-11T20:43:22.831 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.827+0000 7f58d466fec0 20 started sync module instance, tier type = 2019-12-11T20:43:22.831 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:22.827+0000 7f58d466fec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-11T20:43:23.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.180+0000 7f58c1899700 0 -- 172.21.15.79:0/1783892691 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x55635dfdcda0 msgr2=0x55635dfdf270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T20:43:23.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.381+0000 7f58d466fec0 20 add_watcher() i=0 2019-12-11T20:43:23.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.381+0000 7f58d466fec0 20 add_watcher() i=1 2019-12-11T20:43:23.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.381+0000 7f58d466fec0 20 add_watcher() i=2 2019-12-11T20:43:23.385 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.381+0000 7f58d466fec0 20 add_watcher() i=3 2019-12-11T20:43:23.386 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.381+0000 7f58d466fec0 20 add_watcher() i=4 2019-12-11T20:43:23.386 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.381+0000 7f58d466fec0 20 add_watcher() i=5 2019-12-11T20:43:23.386 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.381+0000 7f58d466fec0 20 add_watcher() i=6 2019-12-11T20:43:23.423 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.420+0000 7f58d466fec0 20 add_watcher() i=7 2019-12-11T20:43:23.424 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.420+0000 7f58d466fec0 2 all 8 watchers are set, enabling cache 2019-12-11T20:43:23.424 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.420+0000 7f58927dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-11T20:43:23.449 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:23.445+0000 7f58b9098700 0 -- 172.21.15.79:0/3607920993 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x55635dff6580 msgr2=0x55635dff8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-11T20:43:25.041 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.037+0000 7f5808ff9700 20 reqs_thread_entry: start 2019-12-11T20:43:25.041 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.038+0000 7f58027fc700 20 reqs_thread_entry: start 2019-12-11T20:43:25.042 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.038+0000 7f58d466fec0 20 init_complete bucket index max shards: 0 2019-12-11T20:43:25.042 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.038+0000 7f5800ff9700 20 reqs_thread_entry: start 2019-12-11T20:43:25.042 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.038+0000 7f58d466fec0 20 get_system_obj_state: rctx=0x7ffd338cd668 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55635e06b590 s->prefetch_data=0 2019-12-11T20:43:25.042 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.038+0000 7f58d466fec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-11T20:43:25.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.051+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2019-12-11T20:43:25.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.051+0000 7f58d466fec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T20:43:25.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.051+0000 7f58d466fec0 20 get_system_obj_state: rctx=0x7ffd338cd678 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55635e06d320 s->prefetch_data=0 2019-12-11T20:43:25.055 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.051+0000 7f58d466fec0 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2019-12-11T20:43:25.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.058+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2019-12-11T20:43:25.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.058+0000 7f58d466fec0 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-11T20:43:25.061 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.058+0000 7f58d466fec0 20 get_system_obj_state: rctx=0x7ffd338cd288 obj=default.rgw.meta:users.swift:bar.client.0:bar state=0x55635e06d320 s->prefetch_data=0 2019-12-11T20:43:25.062 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.058+0000 7f58d466fec0 10 cache get: name=default.rgw.meta+users.swift+bar.client.0:bar : miss 2019-12-11T20:43:25.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.064+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x0 2019-12-11T20:43:25.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.064+0000 7f58d466fec0 10 adding default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-11T20:43:25.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.064+0000 7f58d466fec0 20 get_system_obj_state: rctx=0x7ffd338cd408 obj=default.rgw.meta:users.swift:bar.client.0:bar state=0x55635e06d5d0 s->prefetch_data=0 2019-12-11T20:43:25.068 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.064+0000 7f58d466fec0 10 cache get: name=default.rgw.meta+users.swift+bar.client.0:bar : type miss (requested=0x6, cached=0x0) 2019-12-11T20:43:25.073 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.070+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x0 2019-12-11T20:43:25.073 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.070+0000 7f58d466fec0 10 moving default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-11T20:43:25.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.127+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T20:43:25.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.127+0000 7f58d466fec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T20:43:25.131 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.127+0000 7f58d466fec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=612 2019-12-11T20:43:25.143 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.140+0000 7f58a2ffd700 10 RGWWatcher::handle_notify() notify_id 592705486915 cookie 93885267008912 notifier 4694 bl.length()=612 2019-12-11T20:43:25.143 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.140+0000 7f58a2ffd700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T20:43:25.144 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.140+0000 7f58a2ffd700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T20:43:25.452 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.448+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-11T20:43:25.452 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.448+0000 7f58d466fec0 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-11T20:43:25.452 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.448+0000 7f58d466fec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=194 2019-12-11T20:43:25.464 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.460+0000 7f58a2ffd700 10 RGWWatcher::handle_notify() notify_id 592705486916 cookie 93885267008912 notifier 4694 bl.length()=194 2019-12-11T20:43:25.465 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.460+0000 7f58a2ffd700 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-11T20:43:25.465 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.460+0000 7f58a2ffd700 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-11T20:43:25.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.568+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x7 2019-12-11T20:43:25.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.568+0000 7f58d466fec0 10 moving default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-11T20:43:25.572 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.568+0000 7f58d466fec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=183 2019-12-11T20:43:25.580 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.577+0000 7f58a2ffd700 10 RGWWatcher::handle_notify() notify_id 592705487818 cookie 93885267030464 notifier 4694 bl.length()=183 2019-12-11T20:43:25.580 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.577+0000 7f58a2ffd700 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x7 2019-12-11T20:43:25.581 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.577+0000 7f58a2ffd700 10 moving default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-11T20:43:25.706 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.703+0000 7f58d466fec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T20:43:25.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.703+0000 7f58d466fec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T20:43:25.707 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.703+0000 7f58d466fec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=636 2019-12-11T20:43:25.724 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.720+0000 7f58a2ffd700 10 RGWWatcher::handle_notify() notify_id 592705486917 cookie 93885267008912 notifier 4694 bl.length()=636 2019-12-11T20:43:25.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.720+0000 7f58a2ffd700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-11T20:43:25.725 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:25.720+0000 7f58a2ffd700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-11T20:43:26.371 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2019-12-11T20:43:26.371 INFO:teuthology.orchestra.run.smithi079.stdout: "user_id": "bar.client.0", 2019-12-11T20:43:26.371 INFO:teuthology.orchestra.run.smithi079.stdout: "display_name": "Mr. bar.client.0 bar", 2019-12-11T20:43:26.371 INFO:teuthology.orchestra.run.smithi079.stdout: "email": "bar.client.0+test@test.test", 2019-12-11T20:43:26.371 INFO:teuthology.orchestra.run.smithi079.stdout: "suspended": 0, 2019-12-11T20:43:26.372 INFO:teuthology.orchestra.run.smithi079.stdout: "max_buckets": 1000, 2019-12-11T20:43:26.372 INFO:teuthology.orchestra.run.smithi079.stdout: "subusers": [ 2019-12-11T20:43:26.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T20:43:26.372 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "bar.client.0:bar", 2019-12-11T20:43:26.372 INFO:teuthology.orchestra.run.smithi079.stdout: "permissions": "full-control" 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: "keys": [], 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: "swift_keys": [ 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: "user": "bar.client.0:bar", 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "DO0GrnskfuSWBt+yWdfYDDC9yZ3tzBfvzUhxGh1HDr7ZDzcBEg2TdQ==" 2019-12-11T20:43:26.373 INFO:teuthology.orchestra.run.smithi079.stdout: } 2019-12-11T20:43:26.374 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2019-12-11T20:43:26.374 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": [], 2019-12-11T20:43:26.374 INFO:teuthology.orchestra.run.smithi079.stdout: "op_mask": "read, write, delete", 2019-12-11T20:43:26.374 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "", 2019-12-11T20:43:26.374 INFO:teuthology.orchestra.run.smithi079.stdout: "default_storage_class": "", 2019-12-11T20:43:26.374 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_tags": [], 2019-12-11T20:43:26.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2019-12-11T20:43:26.375 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T20:43:26.375 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T20:43:26.375 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T20:43:26.375 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T20:43:26.375 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T20:43:26.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T20:43:26.375 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2019-12-11T20:43:26.376 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2019-12-11T20:43:26.376 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2019-12-11T20:43:26.376 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2019-12-11T20:43:26.376 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2019-12-11T20:43:26.376 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2019-12-11T20:43:26.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2019-12-11T20:43:26.376 INFO:teuthology.orchestra.run.smithi079.stdout: "temp_url_keys": [], 2019-12-11T20:43:26.377 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "rgw", 2019-12-11T20:43:26.377 INFO:teuthology.orchestra.run.smithi079.stdout: "mfa_ids": [] 2019-12-11T20:43:26.377 INFO:teuthology.orchestra.run.smithi079.stdout:} 2019-12-11T20:43:26.377 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T20:43:26.441 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.438+0000 7f58d466fec0 20 remove_watcher() i=0 2019-12-11T20:43:26.442 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.438+0000 7f58d466fec0 2 removed watcher, disabling cache 2019-12-11T20:43:26.505 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.502+0000 7f58d466fec0 20 remove_watcher() i=1 2019-12-11T20:43:26.571 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.567+0000 7f58d466fec0 20 remove_watcher() i=2 2019-12-11T20:43:26.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.631+0000 7f58d466fec0 20 remove_watcher() i=3 2019-12-11T20:43:26.691 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.687+0000 7f58d466fec0 20 remove_watcher() i=4 2019-12-11T20:43:26.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.742+0000 7f58d466fec0 20 remove_watcher() i=5 2019-12-11T20:43:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.810+0000 7f58d466fec0 20 remove_watcher() i=6 2019-12-11T20:43:26.881 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-11T20:43:26.878+0000 7f58d466fec0 20 remove_watcher() i=7 2019-12-11T20:43:26.892 INFO:tasks.swift:Configuring testswift... 2019-12-11T20:43:26.893 INFO:teuthology.orchestra.run.smithi079:> cd /home/ubuntu/cephtest/swift && ./bootstrap 2019-12-11T20:43:29.472 INFO:teuthology.orchestra.run.smithi079.stdout:New python executable in /home/ubuntu/cephtest/swift/virtualenv/bin/python 2019-12-11T20:43:30.727 INFO:teuthology.orchestra.run.smithi079.stdout:Installing setuptools, pip, wheel...done. 2019-12-11T20:43:30.985 INFO:teuthology.orchestra.run.smithi079.stderr:Cache entry deserialization failed, entry ignored 2019-12-11T20:43:31.112 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting pip 2019-12-11T20:43:31.191 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl 2019-12-11T20:43:31.258 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: pip 2019-12-11T20:43:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: pip 9.0.1 2019-12-11T20:43:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: Uninstalling pip-9.0.1: 2019-12-11T20:43:31.363 INFO:teuthology.orchestra.run.smithi079.stdout: Successfully uninstalled pip-9.0.1 2019-12-11T20:43:31.966 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed pip-19.3.1 2019-12-11T20:43:32.277 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T20:43:32.939 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting setuptools==32.3.1 2019-12-11T20:43:32.948 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/69/19/b1dff551058ce79d88b1e3688f1c735590d7ddf44d10681512133b35019f/setuptools-32.3.1-py2.py3-none-any.whl 2019-12-11T20:43:33.002 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: setuptools 2019-12-11T20:43:33.002 INFO:teuthology.orchestra.run.smithi079.stdout: Found existing installation: setuptools 28.8.0 2019-12-11T20:43:33.027 INFO:teuthology.orchestra.run.smithi079.stdout: Uninstalling setuptools-28.8.0: 2019-12-11T20:43:33.041 INFO:teuthology.orchestra.run.smithi079.stdout: Successfully uninstalled setuptools-28.8.0 2019-12-11T20:43:33.232 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed setuptools-32.3.1 2019-12-11T20:43:33.817 INFO:teuthology.orchestra.run.smithi079.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-11T20:43:33.995 INFO:teuthology.orchestra.run.smithi079.stdout:Processing /home/ubuntu/.cache/pip/wheels/54/b7/c7/2ada654ee54483c9329871665aaf4a6056c3ce36f29cf66e67/PyYAML-5.2-cp27-cp27mu-linux_x86_64.whl 2019-12-11T20:43:34.048 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting nose>=1.0.0 2019-12-11T20:43:34.049 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl 2019-12-11T20:43:34.174 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting boto>=2.0b4 2019-12-11T20:43:34.180 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl 2019-12-11T20:43:34.331 INFO:teuthology.orchestra.run.smithi079.stdout:Processing /home/ubuntu/.cache/pip/wheels/56/0f/19/fbbf81e5764e6d8b74501c4357a88c14c94466ec777c03734c/bunch-1.0.1-cp27-none-any.whl 2019-12-11T20:43:34.574 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting gevent>=0.14 2019-12-11T20:43:34.588 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/e6/e9/3a693414f706e12abe60554cd73c5ae8f848b182ae58018f93d86c9eb418/gevent-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-11T20:43:34.844 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting isodate>=0.4.4 2019-12-11T20:43:34.846 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl 2019-12-11T20:43:34.925 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting eventlet>=0.9.14 2019-12-11T20:43:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/ef/01/83454d11bac9015f34e93cf11efcae169390ddf2df97cb73ca86de6465ed/eventlet-0.25.1-py2.py3-none-any.whl (222kB) 2019-12-11T20:43:35.161 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting netifaces>=0.5 2019-12-11T20:43:35.170 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/7e/02/ad1a92a72620cc17d448fe4dbdfbdf8fe1487ee7bfd82bb48308712c2f3c/netifaces-0.10.9-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-11T20:43:35.252 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting xattr<0.7.0,>=0.6 2019-12-11T20:43:35.262 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/c5/80/b25d549ae4bf4f3e9635a331b759ffca2de4dd8a78dc5106d1ca92f5d08d/xattr-0.6.4.tar.gz 2019-12-11T20:43:35.672 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting simplejson>=2.1.3 2019-12-11T20:43:35.685 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/98/87/a7b98aa9256c8843f92878966dc3d8d914c14aad97e2c5ce4798d5743e07/simplejson-3.17.0.tar.gz (83kB) 2019-12-11T20:43:36.034 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting WebOb==1.0 2019-12-11T20:43:36.045 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/bc/eb/91b8dd8f3a8d3c3943bf27a2d322501ad8b227bf29f59e792c2eae092bc9/WebOb-1.0.tar.gz (210kB) 2019-12-11T20:43:36.479 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" 2019-12-11T20:43:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/8b/6e/f2d25875713ad0885c8d3c69269697406652e6f64e1a6bd8264f7a609327/greenlet-0.4.15-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-11T20:43:36.568 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting six 2019-12-11T20:43:36.572 INFO:teuthology.orchestra.run.smithi079.stdout: Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl 2019-12-11T20:43:36.648 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting monotonic>=1.4 2019-12-11T20:43:36.658 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/ac/aa/063eca6a416f397bd99552c534c6d11d57f58f2e94c14780f3bbf818c4cf/monotonic-1.5-py2.py3-none-any.whl 2019-12-11T20:43:36.753 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting dnspython>=1.15.0 2019-12-11T20:43:36.763 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl (188kB) 2019-12-11T20:43:36.887 INFO:teuthology.orchestra.run.smithi079.stdout:Collecting enum34; python_version < "3.4" 2019-12-11T20:43:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl 2019-12-11T20:43:36.936 INFO:teuthology.orchestra.run.smithi079.stdout:Building wheels for collected packages: xattr, simplejson, WebOb 2019-12-11T20:43:36.936 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for xattr (setup.py): started 2019-12-11T20:43:37.383 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for xattr (setup.py): finished with status 'done' 2019-12-11T20:43:37.383 INFO:teuthology.orchestra.run.smithi079.stdout: Created wheel for xattr: filename=xattr-0.6.4-cp27-cp27mu-linux_x86_64.whl size=24121 sha256=ef0cac768c74281c32ec5f304e0a09511e2cecd79b0e63a65afc94a8ac52f742 2019-12-11T20:43:37.384 INFO:teuthology.orchestra.run.smithi079.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/cb/2c/40/ea48345a21b521cdacebf61db13e3bf0c42c77ec664e3fe9e5 2019-12-11T20:43:37.390 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for simplejson (setup.py): started 2019-12-11T20:43:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for simplejson (setup.py): finished with status 'done' 2019-12-11T20:43:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: Created wheel for simplejson: filename=simplejson-3.17.0-cp27-cp27mu-linux_x86_64.whl size=113130 sha256=f2844f3f67162eb3df960342e4689c5094f007626151a90c4733051615cf6042 2019-12-11T20:43:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/86/c0/83/dcd0339abb2640544bb8e0938aab2d069cef55e5647ce6e097 2019-12-11T20:43:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for WebOb (setup.py): started 2019-12-11T20:43:38.496 INFO:teuthology.orchestra.run.smithi079.stdout: Building wheel for WebOb (setup.py): finished with status 'done' 2019-12-11T20:43:38.498 INFO:teuthology.orchestra.run.smithi079.stdout: Created wheel for WebOb: filename=WebOb-1.0-cp27-none-any.whl size=58478 sha256=d9675c4d2fb3215ad709685c7d0442b3a032db411e58f8888c2d4270edc9cde5 2019-12-11T20:43:38.499 INFO:teuthology.orchestra.run.smithi079.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/30/41/c0/8ff9e679d72e05f355e6e2ec87d026131538c953a3b12496eb 2019-12-11T20:43:38.504 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully built xattr simplejson WebOb 2019-12-11T20:43:38.549 INFO:teuthology.orchestra.run.smithi079.stdout:Installing collected packages: PyYAML, nose, boto, bunch, greenlet, gevent, six, isodate, monotonic, dnspython, enum34, eventlet, netifaces, xattr, simplejson, WebOb 2019-12-11T20:43:40.097 INFO:teuthology.orchestra.run.smithi079.stdout:Successfully installed PyYAML-5.2 WebOb-1.0 boto-2.49.0 bunch-1.0.1 dnspython-1.16.0 enum34-1.1.6 eventlet-0.25.1 gevent-1.4.0 greenlet-0.4.15 isodate-0.6.0 monotonic-1.5 netifaces-0.10.9 nose-1.3.7 simplejson-3.17.0 six-1.13.0 xattr-0.6.4 2019-12-11T20:43:40.411 INFO:teuthology.orchestra.run.smithi079.stdout:running develop 2019-12-11T20:43:40.463 INFO:teuthology.orchestra.run.smithi079.stdout:running egg_info 2019-12-11T20:43:40.463 INFO:teuthology.orchestra.run.smithi079.stdout:creating swift.egg-info 2019-12-11T20:43:40.464 INFO:teuthology.orchestra.run.smithi079.stdout:writing swift.egg-info/PKG-INFO 2019-12-11T20:43:40.464 INFO:teuthology.orchestra.run.smithi079.stdout:writing top-level names to swift.egg-info/top_level.txt 2019-12-11T20:43:40.464 INFO:teuthology.orchestra.run.smithi079.stdout:writing dependency_links to swift.egg-info/dependency_links.txt 2019-12-11T20:43:40.468 INFO:teuthology.orchestra.run.smithi079.stdout:writing entry points to swift.egg-info/entry_points.txt 2019-12-11T20:43:40.468 INFO:teuthology.orchestra.run.smithi079.stdout:writing manifest file 'swift.egg-info/SOURCES.txt' 2019-12-11T20:43:40.473 INFO:teuthology.orchestra.run.smithi079.stdout:reading manifest file 'swift.egg-info/SOURCES.txt' 2019-12-11T20:43:40.473 INFO:teuthology.orchestra.run.smithi079.stdout:reading manifest template 'MANIFEST.in' 2019-12-11T20:43:40.473 INFO:teuthology.orchestra.run.smithi079.stderr:warning: no files found matching 'ChangeLog' 2019-12-11T20:43:40.476 INFO:teuthology.orchestra.run.smithi079.stdout:writing manifest file 'swift.egg-info/SOURCES.txt' 2019-12-11T20:43:40.477 INFO:teuthology.orchestra.run.smithi079.stdout:running build_ext 2019-12-11T20:43:40.477 INFO:teuthology.orchestra.run.smithi079.stdout:Creating /home/ubuntu/cephtest/swift/virtualenv/lib/python2.7/site-packages/swift.egg-link (link to .) 2019-12-11T20:43:40.477 INFO:teuthology.orchestra.run.smithi079.stdout:Adding swift 1.4.4 to easy-install.pth file 2019-12-11T20:43:40.494 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.495 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-account-auditor script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.497 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-account-audit script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.498 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-account-reaper script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.499 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-account-replicator script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.499 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-account-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.500 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-container-auditor script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.501 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-container-replicator script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.502 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-container-sync script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.502 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-container-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.504 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-container-updater script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.505 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-drive-audit script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.506 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-get-nodes script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.507 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-init script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.508 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-object-auditor script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.508 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-object-expirer script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.509 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-object-info script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.510 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-object-replicator script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.511 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-object-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.512 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-object-updater script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.513 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-proxy-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.516 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-ring-builder script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.518 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-stats-populate script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.524 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-stats-report script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.525 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-dispersion-populate script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.527 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-dispersion-report script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.528 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-bench script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.531 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-recon script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.532 INFO:teuthology.orchestra.run.smithi079.stdout:Installing swift-recon-cron script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-11T20:43:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: 2019-12-11T20:43:40.532 INFO:teuthology.orchestra.run.smithi079.stdout:Installed /home/ubuntu/cephtest/swift 2019-12-11T20:43:40.533 INFO:teuthology.orchestra.run.smithi079.stdout:Processing dependencies for swift==1.4.4 2019-12-11T20:43:40.533 INFO:teuthology.orchestra.run.smithi079.stdout:Finished processing dependencies for swift==1.4.4 2019-12-11T20:43:40.545 INFO:teuthology.orchestra.run.smithi079:> cat > /home/ubuntu/cephtest/archive/testswift.client.0.conf 2019-12-11T20:43:40.589 INFO:teuthology.orchestra.run.smithi079:> SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw' 2019-12-11T20:44:03.708 INFO:teuthology.orchestra.run.smithi079.stderr:testAccountHead (test.functional.tests.TestAccount) ... ok 2019-12-11T20:44:08.394 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:11:55.324 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.437 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:11:55.362 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.469 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:11:55.370 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.471 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:11:55.401 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.473 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:11:55.403 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.478 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:11:55.380 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.498 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:11:55.429 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.536 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:11:55.466 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:08.560 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:11:55.461 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:44:12.131 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerListing (test.functional.tests.TestAccount) ... ok 2019-12-11T20:47:23.938 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:15:10.869 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:23.944 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:15:10.875 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:23.954 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:15:10.884 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:23.956 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:15:10.886 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:23.964 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:15:10.866 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:23.974 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:15:10.876 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:24.073 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:15:11.004 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:24.082 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:15:11.013 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:24.133 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:15:11.035 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:47:27.071 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerSerializedInfo (test.functional.tests.TestAccount) ... ok 2019-12-11T20:47:37.051 INFO:teuthology.orchestra.run.smithi079.stderr:testContainersOrderedByName (test.functional.tests.TestAccount) ... ok 2019-12-11T20:47:38.551 INFO:teuthology.orchestra.run.smithi079.stderr:testInvalidAuthToken (test.functional.tests.TestAccount) ... ok 2019-12-11T20:47:47.609 INFO:teuthology.orchestra.run.smithi079.stderr:testLastContainerMarker (test.functional.tests.TestAccount) ... ok 2019-12-11T20:48:01.792 INFO:teuthology.orchestra.run.smithi079.stderr:testListingLimit (test.functional.tests.TestAccount) ... ok 2019-12-11T20:48:31.975 INFO:teuthology.orchestra.run.smithi079.stderr:testMarkerLimitContainerList (test.functional.tests.TestAccount) ... ok 2019-12-11T20:48:32.019 INFO:teuthology.orchestra.run.smithi079.stderr:testPUT (test.functional.tests.TestAccount) ... ok 2019-12-11T20:52:34.969 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:20:21.899 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:34.974 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:20:21.905 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:34.983 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:20:21.913 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:34.985 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:20:21.887 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:34.986 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:20:21.916 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:34.991 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:20:21.893 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:35.171 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:20:22.101 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:35.174 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:20:22.105 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:35.189 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:20:22.091 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.897 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:20:26.828 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.913 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:20:26.843 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.919 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:20:26.850 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.935 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:20:26.865 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.968 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:20:26.898 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.968 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:20:26.870 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.976 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:20:26.877 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.991 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:20:26.893 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:52:39.997 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:20:26.927 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:53:38.226 INFO:teuthology.orchestra.run.smithi079.stderr:testGetRequest (test.functional.tests.TestAccountNoContainers) ... ok 2019-12-11T20:53:40.182 INFO:teuthology.orchestra.run.smithi079.stderr:testGetRequest (test.functional.tests.TestAccountNoContainersUTF8) ... ok 2019-12-11T20:54:04.911 INFO:teuthology.orchestra.run.smithi079.stderr:testAccountHead (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:54:15.384 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerListing (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:56:15.682 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:24:02.613 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.719 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:24:02.650 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.737 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:24:02.667 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.741 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:24:02.669 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.759 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:24:02.661 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.765 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:24:02.667 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.893 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:24:02.824 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.957 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:24:02.859 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:56:15.959 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:24:02.890 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.076 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:25:33.006 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.082 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:25:33.013 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.096 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:25:33.026 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.103 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:25:33.033 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.116 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:25:33.015 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.126 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:25:33.028 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.171 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:25:33.102 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.184 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:25:33.111 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:46.232 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:25:33.134 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T20:57:55.833 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerSerializedInfo (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:58:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:testContainersOrderedByName (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:58:08.394 INFO:teuthology.orchestra.run.smithi079.stderr:testInvalidAuthToken (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:58:17.993 INFO:teuthology.orchestra.run.smithi079.stderr:testLastContainerMarker (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:58:34.180 INFO:teuthology.orchestra.run.smithi079.stderr:testListingLimit (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:59:55.690 INFO:teuthology.orchestra.run.smithi079.stderr:testMarkerLimitContainerList (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T20:59:55.754 INFO:teuthology.orchestra.run.smithi079.stderr:testPUT (test.functional.tests.TestAccountUTF8) ... ok 2019-12-11T21:01:01.693 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:28:48.621 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.699 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:28:48.630 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.716 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:28:48.645 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.723 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:28:48.653 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.740 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:28:48.642 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.747 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:28:48.648 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.798 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:28:48.727 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.821 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:28:48.745 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:01:01.885 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:28:48.785 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.257 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:30:43.187 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.262 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:30:43.192 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.272 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:30:43.202 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.275 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:30:43.206 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.282 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:30:43.184 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.289 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:30:43.191 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.363 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:30:43.294 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.389 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:30:43.319 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:02:56.415 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:30:43.317 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:05:42.446 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerExistenceCachingProblem (test.functional.tests.TestContainer) ... ok 2019-12-11T21:05:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerFileList (test.functional.tests.TestContainer) ... ok 2019-12-11T21:05:44.129 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerFileListOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-11T21:05:45.520 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerFileListWithLimit (test.functional.tests.TestContainer) ... ok 2019-12-11T21:05:45.939 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerInfo (test.functional.tests.TestContainer) ... ok 2019-12-11T21:05:46.254 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerInfoOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-11T21:05:53.793 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerNameLimit (test.functional.tests.TestContainer) ... ok 2019-12-11T21:05:58.868 INFO:teuthology.orchestra.run.smithi079.stderr:testCreate (test.functional.tests.TestContainer) ... ok 2019-12-11T21:06:08.421 INFO:teuthology.orchestra.run.smithi079.stderr:testDelete (test.functional.tests.TestContainer) ... ok 2019-12-11T21:06:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-11T21:06:12.383 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteOnContainerWithFiles (test.functional.tests.TestContainer) ... ok 2019-12-11T21:06:12.422 INFO:teuthology.orchestra.run.smithi079.stderr:testFileCreateInContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-11T21:07:12.925 INFO:teuthology.orchestra.run.smithi079.stderr:testFileListingLimitMarkerPrefix (test.functional.tests.TestContainer) ... ok 2019-12-11T21:07:15.191 INFO:teuthology.orchestra.run.smithi079.stderr:testFileOrder (test.functional.tests.TestContainer) ... ok 2019-12-11T21:07:28.441 INFO:teuthology.orchestra.run.smithi079.stderr:testFileThenContainerDelete (test.functional.tests.TestContainer) ... ok 2019-12-11T21:07:32.369 INFO:teuthology.orchestra.run.smithi079.stderr:testLastFileMarker (test.functional.tests.TestContainer) ... ok 2019-12-11T21:07:57.516 INFO:teuthology.orchestra.run.smithi079.stderr:testMarkerLimitFileList (test.functional.tests.TestContainer) ... ok 2019-12-11T21:08:07.944 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:35:54.875 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:07.948 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:35:54.879 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:07.971 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:35:54.873 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:07.980 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:35:54.910 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:07.985 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:35:54.916 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:07.992 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:35:54.894 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:08.181 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:35:55.111 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:08.185 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:35:55.116 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:08.241 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:35:55.143 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:08:37.832 INFO:teuthology.orchestra.run.smithi079.stderr:testPrefixAndLimit (test.functional.tests.TestContainer) ... ok 2019-12-11T21:08:37.911 INFO:teuthology.orchestra.run.smithi079.stderr:testTooLongName (test.functional.tests.TestContainer) ... ok 2019-12-11T21:09:23.101 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:37:10.031 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.142 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:37:10.072 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.152 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:37:10.083 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.175 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:37:10.106 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.179 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:37:10.082 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.195 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:37:10.097 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.209 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:37:10.140 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.234 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:37:10.164 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:23.255 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:37:10.157 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:09:27.133 INFO:teuthology.orchestra.run.smithi079.stderr:testUnorderedList (test.functional.tests.TestContainer) ... ok 2019-12-11T21:13:00.104 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerListing (test.functional.tests.TestContainerPaths) ... ok 2019-12-11T21:13:06.452 INFO:teuthology.orchestra.run.smithi079.stderr:testStructure (test.functional.tests.TestContainerPaths) ... ok 2019-12-11T21:13:18.246 INFO:teuthology.orchestra.run.smithi079.stderr:testTraverseContainer (test.functional.tests.TestContainerPaths) ... ok 2019-12-11T21:13:18.571 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:41:05.501 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.579 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:41:05.510 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.591 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:41:05.521 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.596 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:41:05.498 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.597 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:41:05.528 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.616 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:41:05.518 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.733 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:41:05.664 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.764 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:41:05.694 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:13:18.835 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:41:05.737 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:14:28.760 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerExistenceCachingProblem (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:30.544 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerFileList (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:31.429 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerFileListOnContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:32.881 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerFileListWithLimit (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:33.005 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerInfo (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:33.386 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerInfoOnContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:38.922 INFO:teuthology.orchestra.run.smithi079.stderr:testContainerNameLimit (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:41.970 INFO:teuthology.orchestra.run.smithi079.stderr:testCreate (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:50.502 INFO:teuthology.orchestra.run.smithi079.stderr:testDelete (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:50.617 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteOnContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:54.974 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteOnContainerWithFiles (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:14:55.099 INFO:teuthology.orchestra.run.smithi079.stderr:testFileCreateInContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:15:47.101 INFO:teuthology.orchestra.run.smithi079.stderr:testFileListingLimitMarkerPrefix (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:15:48.594 INFO:teuthology.orchestra.run.smithi079.stderr:testFileOrder (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:16:00.801 INFO:teuthology.orchestra.run.smithi079.stderr:testFileThenContainerDelete (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:16:04.427 INFO:teuthology.orchestra.run.smithi079.stderr:testLastFileMarker (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:16:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:testMarkerLimitFileList (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:17:18.883 INFO:teuthology.orchestra.run.smithi079.stderr:testPrefixAndLimit (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:17:18.922 INFO:teuthology.orchestra.run.smithi079.stderr:testTooLongName (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:17:44.913 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:45:31.844 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:44.923 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:45:31.853 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:44.935 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:45:31.864 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:44.939 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:45:31.869 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:44.942 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:45:31.845 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:44.947 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:45:31.849 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:45.049 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:45:31.980 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:45.059 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:45:31.962 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:17:45.179 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:45:32.109 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:08.780 INFO:teuthology.orchestra.run.smithi079.stderr:testUnorderedList (test.functional.tests.TestContainerUTF8) ... ok 2019-12-11T21:18:30.107 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:46:17.038 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.113 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:46:17.044 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.139 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:46:17.042 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.148 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:46:17.051 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.156 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:46:17.087 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.161 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:46:17.092 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.167 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:46:17.098 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.222 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:46:17.152 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:18:30.276 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:46:17.178 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:21:32.214 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_manifest (test.functional.tests.TestDlo) ... ok 2019-12-11T21:21:32.577 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_manifest_document_itself (test.functional.tests.TestDlo) ... ok 2019-12-11T21:21:36.866 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_range (test.functional.tests.TestDlo) ... ok 2019-12-11T21:21:37.108 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_range_out_of_range (test.functional.tests.TestDlo) ... ok 2019-12-11T21:22:25.790 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_manifest (test.functional.tests.TestDloUTF8) ... ok 2019-12-11T21:22:26.530 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_manifest_document_itself (test.functional.tests.TestDloUTF8) ... ok 2019-12-11T21:22:31.023 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_range (test.functional.tests.TestDloUTF8) ... ok 2019-12-11T21:22:31.762 INFO:teuthology.orchestra.run.smithi079.stderr:test_get_range_out_of_range (test.functional.tests.TestDloUTF8) ... ok 2019-12-11T21:22:57.235 INFO:teuthology.orchestra.run.smithi079.stderr:testBadHeaders (test.functional.tests.TestFile) ... ok 2019-12-11T21:22:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:testBlankMetadataName (test.functional.tests.TestFile) ... ok 2019-12-11T21:23:42.903 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:51:29.834 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:42.909 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:51:29.840 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:42.915 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:51:29.846 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:42.918 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:51:29.849 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:42.925 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:51:29.826 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:42.929 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:51:29.832 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:43.099 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:51:30.029 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:43.121 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:51:30.023 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:23:43.124 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:51:30.056 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.377 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:53:53.308 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.381 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:53:53.311 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.407 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:53:53.338 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.443 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:53:53.374 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.449 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:53:53.379 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.472 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:53:53.374 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.493 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:53:53.396 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.522 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:53:53.453 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:26:06.606 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:53:53.509 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.057 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:56:39.988 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.063 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:56:39.994 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.070 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:56:40.001 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.074 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:56:39.976 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.074 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:56:40.005 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.078 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:56:39.981 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.179 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:04:56:40.110 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.189 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:04:56:40.119 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:28:53.203 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:04:56:40.106 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:04.800 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:01:51.731 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:04.804 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:01:51.735 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:04.828 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:01:51.757 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:04.829 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:01:51.760 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:04.830 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:01:51.732 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:04.840 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:01:51.742 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:05.006 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:01:51.937 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:05.010 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:01:51.941 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:05.112 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:01:52.014 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.782 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:02:14.713 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.807 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:02:14.737 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.807 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:02:14.738 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.831 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:02:14.762 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.836 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:02:14.767 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.844 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:02:14.746 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.846 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:02:14.748 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.860 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:02:14.790 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:34:27.875 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:02:14.770 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:21.196 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:35:21.185+0000 1694f700 -1 osd.1 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:37.065228+0000 front 2019-12-11T21:34:41.122412+0000 (oldest deadline 2019-12-11T21:35:20.969795+0000) 2019-12-11T21:35:21.881 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:35:21.868+0000 1694f700 -1 osd.6 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:41.121860+0000 front 2019-12-11T21:34:36.551531+0000 (oldest deadline 2019-12-11T21:35:20.995887+0000) 2019-12-11T21:35:22.225 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:35:22.215+0000 1694f700 -1 osd.1 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:37.065228+0000 front 2019-12-11T21:34:41.122412+0000 (oldest deadline 2019-12-11T21:35:20.969795+0000) 2019-12-11T21:35:22.398 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:35:22.381+0000 1694f700 -1 osd.2 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:40.331970+0000 front 2019-12-11T21:34:40.807342+0000 (oldest deadline 2019-12-11T21:35:22.003250+0000) 2019-12-11T21:35:22.451 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:35:22.441+0000 1694f700 -1 osd.3 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:36.771585+0000 front 2019-12-11T21:34:39.211234+0000 (oldest deadline 2019-12-11T21:35:21.536831+0000) 2019-12-11T21:35:22.764 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:35:22.755+0000 1694f700 -1 osd.0 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:40.755430+0000 front 2019-12-11T21:34:39.211509+0000 (oldest deadline 2019-12-11T21:35:22.249744+0000) 2019-12-11T21:35:22.879 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:35:22.863+0000 1694f700 -1 osd.6 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:41.121860+0000 front 2019-12-11T21:34:36.551531+0000 (oldest deadline 2019-12-11T21:35:20.995887+0000) 2019-12-11T21:35:23.245 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:35:23.238+0000 1694f700 -1 osd.1 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:37.065228+0000 front 2019-12-11T21:34:41.122412+0000 (oldest deadline 2019-12-11T21:35:20.969795+0000) 2019-12-11T21:35:23.373 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:35:23.367+0000 1694f700 -1 osd.2 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:40.331970+0000 front 2019-12-11T21:34:40.807342+0000 (oldest deadline 2019-12-11T21:35:22.003250+0000) 2019-12-11T21:35:23.434 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:35:23.422+0000 1694f700 -1 osd.3 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:36.771585+0000 front 2019-12-11T21:34:39.211234+0000 (oldest deadline 2019-12-11T21:35:21.536831+0000) 2019-12-11T21:35:23.737 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:35:23.730+0000 1694f700 -1 osd.0 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:40.755430+0000 front 2019-12-11T21:34:39.211509+0000 (oldest deadline 2019-12-11T21:35:22.249744+0000) 2019-12-11T21:35:23.846 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:35:23.840+0000 1694f700 -1 osd.6 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:41.121860+0000 front 2019-12-11T21:34:36.551531+0000 (oldest deadline 2019-12-11T21:35:20.995887+0000) 2019-12-11T21:35:24.279 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:35:24.273+0000 1694f700 -1 osd.1 138 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:34:37.065228+0000 front 2019-12-11T21:34:41.122412+0000 (oldest deadline 2019-12-11T21:35:20.969795+0000) 2019-12-11T21:35:48.254 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:03:35.184 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.288 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:03:35.219 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.336 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:03:35.238 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.344 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:03:35.246 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.371 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:03:35.302 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.408 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:03:35.338 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.425 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:03:35.355 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.481 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:03:35.383 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:35:48.496 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:03:35.427 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:36:21.115 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:36:21.104+0000 1694f700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:37.168814+0000 front 2019-12-11T21:35:37.190940+0000 (oldest deadline 2019-12-11T21:36:21.064179+0000) 2019-12-11T21:36:22.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:36:22.119+0000 1694f700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:37.168814+0000 front 2019-12-11T21:35:37.190940+0000 (oldest deadline 2019-12-11T21:36:21.064179+0000) 2019-12-11T21:36:23.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:36:23.119+0000 1694f700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:37.168814+0000 front 2019-12-11T21:35:37.190940+0000 (oldest deadline 2019-12-11T21:36:21.064179+0000) 2019-12-11T21:36:23.175 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:23.170+0000 1694f700 -1 osd.0 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:40.028866+0000 front 2019-12-11T21:35:40.134502+0000 (oldest deadline 2019-12-11T21:36:23.030638+0000) 2019-12-11T21:36:23.975 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:36:23.968+0000 1694f700 -1 osd.3 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:103579 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821248288 gen 42] snapc 0=[] ondisk+write+known_if_redirected e141) 2019-12-11T21:36:24.082 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:36:24.075+0000 1694f700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:41.057942+0000 front 2019-12-11T21:35:41.045334+0000 (oldest deadline 2019-12-11T21:36:23.907954+0000) 2019-12-11T21:36:24.118 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:36:24.102+0000 1694f700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:37.168814+0000 front 2019-12-11T21:35:37.190940+0000 (oldest deadline 2019-12-11T21:36:21.064179+0000) 2019-12-11T21:36:24.137 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:24.132+0000 1694f700 -1 osd.0 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:40.028866+0000 front 2019-12-11T21:35:40.134502+0000 (oldest deadline 2019-12-11T21:36:23.030638+0000) 2019-12-11T21:36:24.674 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:36:24.668+0000 1694f700 -1 osd.2 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:38.793750+0000 front 2019-12-11T21:35:38.794517+0000 (oldest deadline 2019-12-11T21:36:24.064079+0000) 2019-12-11T21:36:24.971 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:36:24.965+0000 1694f700 -1 osd.3 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:40.670194+0000 front 2019-12-11T21:35:40.744570+0000 (oldest deadline 2019-12-11T21:36:24.470082+0000) 2019-12-11T21:36:24.974 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:36:24.970+0000 1694f700 -1 osd.3 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:103579 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821248288 gen 42] snapc 0=[] ondisk+write+known_if_redirected e141) 2019-12-11T21:36:25.075 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:36:25.069+0000 1694f700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:41.057942+0000 front 2019-12-11T21:35:41.045334+0000 (oldest deadline 2019-12-11T21:36:23.907954+0000) 2019-12-11T21:36:25.091 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:25.085+0000 1694f700 -1 osd.0 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:40.028866+0000 front 2019-12-11T21:35:40.134502+0000 (oldest deadline 2019-12-11T21:36:23.030638+0000) 2019-12-11T21:36:25.115 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:36:25.108+0000 1694f700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:37.168814+0000 front 2019-12-11T21:35:37.190940+0000 (oldest deadline 2019-12-11T21:36:21.064179+0000) 2019-12-11T21:36:25.570 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:36:25.557+0000 1694f700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:40.137948+0000 front 2019-12-11T21:35:40.135811+0000 (oldest deadline 2019-12-11T21:36:24.611164+0000) 2019-12-11T21:36:25.726 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:36:25.720+0000 1694f700 -1 osd.2 142 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:35:38.793750+0000 front 2019-12-11T21:35:38.794517+0000 (oldest deadline 2019-12-11T21:36:24.064079+0000) 2019-12-11T21:36:26.618 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:36:26.566+0000 1694f700 -1 osd.3 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:103579 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821248288 gen 42] snapc 0=[] ondisk+write+known_if_redirected e141) 2019-12-11T21:36:27.775 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T21:36:27.717+0000 20e25700 -1 RGWWatcher::handle_error cookie 821248288 err (107) Transport endpoint is not connected 2019-12-11T21:36:31.221 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:31.209+0000 31f87700 -1 osd.0 145 build_incremental_map_msg missing incremental map 145 2019-12-11T21:36:31.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:31.215+0000 31f87700 -1 osd.0 145 build_incremental_map_msg missing incremental map 145 2019-12-11T21:36:31.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:31.342+0000 35f8f700 -1 osd.0 145 build_incremental_map_msg missing incremental map 145 2019-12-11T21:36:31.901 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:31.828+0000 33f8b700 -1 osd.0 145 build_incremental_map_msg missing incremental map 145 2019-12-11T21:36:31.955 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:31.920+0000 38794700 -1 osd.0 145 build_incremental_map_msg missing incremental map 145 2019-12-11T21:36:55.989 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:55.984+0000 1694f700 -1 osd.0 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:103632 5.5 5:a93a5511:::notify.2:head [watch reconnect cookie 821300688 gen 73] snapc 0=[] ondisk+write+known_if_redirected e143) 2019-12-11T21:36:57.001 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:36:56.997+0000 1694f700 -1 osd.0 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:103632 5.5 5:a93a5511:::notify.2:head [watch reconnect cookie 821300688 gen 73] snapc 0=[] ondisk+write+known_if_redirected e143) 2019-12-11T21:37:29.249 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:37:29.244+0000 38794700 -1 osd.0 147 build_incremental_map_msg missing incremental map 145 2019-12-11T21:37:29.859 INFO:teuthology.orchestra.run.smithi079.stderr:testChunkedPut (test.functional.tests.TestFile) ... ok 2019-12-11T21:37:40.464 INFO:teuthology.orchestra.run.smithi079.stderr:testContentTypeGuessing (test.functional.tests.TestFile) ... ok 2019-12-11T21:38:00.655 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:38:00.617+0000 1694f700 -1 osd.5 150 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected e148) 2019-12-11T21:38:02.639 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:38:02.032+0000 1694f700 -1 osd.5 150 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected e148) 2019-12-11T21:38:25.841 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:06:12.772 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:25.851 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:06:12.782 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:25.855 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:06:12.786 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:25.858 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:06:12.788 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:25.858 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:06:12.760 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:25.867 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:06:12.768 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:26.015 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:06:12.945 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:26.024 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:06:12.926 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:26.086 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:06:13.017 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:38:54.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:38:54.329+0000 1694f700 -1 osd.4 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:11.063444+0000 front 2019-12-11T21:38:11.059850+0000 (oldest deadline 2019-12-11T21:38:53.576452+0000) 2019-12-11T21:38:54.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:38:54.436+0000 1694f700 -1 osd.7 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:13.673506+0000 front 2019-12-11T21:38:11.427501+0000 (oldest deadline 2019-12-11T21:38:53.504910+0000) 2019-12-11T21:38:54.823 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:38:54.819+0000 1694f700 -1 osd.3 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.350436+0000 front 2019-12-11T21:38:10.667596+0000 (oldest deadline 2019-12-11T21:38:54.356443+0000) 2019-12-11T21:38:55.292 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:38:55.285+0000 1694f700 -1 osd.4 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:11.063444+0000 front 2019-12-11T21:38:11.059850+0000 (oldest deadline 2019-12-11T21:38:53.576452+0000) 2019-12-11T21:38:55.396 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:38:55.391+0000 1694f700 -1 osd.7 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:13.673506+0000 front 2019-12-11T21:38:11.427501+0000 (oldest deadline 2019-12-11T21:38:53.504910+0000) 2019-12-11T21:38:55.470 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:38:55.463+0000 1694f700 -1 osd.6 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.703355+0000 front 2019-12-11T21:38:11.100360+0000 (oldest deadline 2019-12-11T21:38:54.468817+0000) 2019-12-11T21:38:55.532 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:38:55.526+0000 1694f700 -1 osd.1 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.008422+0000 front 2019-12-11T21:38:10.010585+0000 (oldest deadline 2019-12-11T21:38:55.152714+0000) 2019-12-11T21:38:55.821 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:38:55.815+0000 1694f700 -1 osd.3 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.350436+0000 front 2019-12-11T21:38:10.667596+0000 (oldest deadline 2019-12-11T21:38:54.356443+0000) 2019-12-11T21:38:56.282 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:38:56.277+0000 1694f700 -1 osd.4 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:11.063444+0000 front 2019-12-11T21:38:11.059850+0000 (oldest deadline 2019-12-11T21:38:53.576452+0000) 2019-12-11T21:38:56.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:38:56.420+0000 1694f700 -1 osd.7 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:13.673506+0000 front 2019-12-11T21:38:11.427501+0000 (oldest deadline 2019-12-11T21:38:53.504910+0000) 2019-12-11T21:38:56.445 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:38:56.440+0000 1694f700 -1 osd.6 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.703355+0000 front 2019-12-11T21:38:11.100360+0000 (oldest deadline 2019-12-11T21:38:54.468817+0000) 2019-12-11T21:38:56.539 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:38:56.535+0000 1694f700 -1 osd.1 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.008422+0000 front 2019-12-11T21:38:10.010585+0000 (oldest deadline 2019-12-11T21:38:55.152714+0000) 2019-12-11T21:38:56.700 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:38:56.694+0000 1694f700 -1 osd.2 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.754705+0000 front 2019-12-11T21:38:10.753435+0000 (oldest deadline 2019-12-11T21:38:56.520535+0000) 2019-12-11T21:38:56.865 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:38:56.860+0000 1694f700 -1 osd.3 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.350436+0000 front 2019-12-11T21:38:10.667596+0000 (oldest deadline 2019-12-11T21:38:54.356443+0000) 2019-12-11T21:38:57.265 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:38:57.256+0000 1694f700 -1 osd.4 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:11.063444+0000 front 2019-12-11T21:38:11.059850+0000 (oldest deadline 2019-12-11T21:38:53.576452+0000) 2019-12-11T21:38:57.442 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:38:57.434+0000 1694f700 -1 osd.6 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.703355+0000 front 2019-12-11T21:38:11.100360+0000 (oldest deadline 2019-12-11T21:38:54.468817+0000) 2019-12-11T21:38:57.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:38:57.441+0000 1694f700 -1 osd.7 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:13.673506+0000 front 2019-12-11T21:38:11.427501+0000 (oldest deadline 2019-12-11T21:38:53.504910+0000) 2019-12-11T21:38:57.550 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:38:57.544+0000 1694f700 -1 osd.1 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.008422+0000 front 2019-12-11T21:38:10.010585+0000 (oldest deadline 2019-12-11T21:38:55.152714+0000) 2019-12-11T21:38:57.655 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:38:57.650+0000 1694f700 -1 osd.2 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.754705+0000 front 2019-12-11T21:38:10.753435+0000 (oldest deadline 2019-12-11T21:38:56.520535+0000) 2019-12-11T21:38:57.906 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:38:57.899+0000 1694f700 -1 osd.3 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.350436+0000 front 2019-12-11T21:38:10.667596+0000 (oldest deadline 2019-12-11T21:38:54.356443+0000) 2019-12-11T21:38:58.257 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:38:58.251+0000 1694f700 -1 osd.4 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:11.063444+0000 front 2019-12-11T21:38:11.059850+0000 (oldest deadline 2019-12-11T21:38:53.576452+0000) 2019-12-11T21:38:58.421 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:38:58.416+0000 1694f700 -1 osd.6 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.703355+0000 front 2019-12-11T21:38:11.100360+0000 (oldest deadline 2019-12-11T21:38:54.468817+0000) 2019-12-11T21:38:58.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:38:58.481+0000 1694f700 -1 osd.7 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:13.673506+0000 front 2019-12-11T21:38:11.427501+0000 (oldest deadline 2019-12-11T21:38:53.504910+0000) 2019-12-11T21:38:58.530 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:38:58.522+0000 1694f700 -1 osd.1 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.008422+0000 front 2019-12-11T21:38:10.010585+0000 (oldest deadline 2019-12-11T21:38:55.152714+0000) 2019-12-11T21:38:58.620 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:38:58.615+0000 1694f700 -1 osd.2 150 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:38:10.754705+0000 front 2019-12-11T21:38:10.753435+0000 (oldest deadline 2019-12-11T21:38:56.520535+0000) 2019-12-11T21:39:29.555 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:39:29.548+0000 1694f700 -1 osd.4 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104008 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 821358992 gen 39] snapc 0=[] ondisk+write+known_if_redirected e151) 2019-12-11T21:39:30.582 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:39:30.575+0000 1694f700 -1 osd.4 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104008 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 821358992 gen 39] snapc 0=[] ondisk+write+known_if_redirected e151) 2019-12-11T21:39:56.969 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:39:56.964+0000 1694f700 -1 osd.2 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:16.069472+0000 front 2019-12-11T21:39:15.150238+0000 (oldest deadline 2019-12-11T21:39:55.978580+0000) 2019-12-11T21:39:57.287 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:39:57.281+0000 1694f700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.167808+0000 front 2019-12-11T21:39:13.181448+0000 (oldest deadline 2019-12-11T21:39:56.638543+0000) 2019-12-11T21:39:57.547 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:39:57.541+0000 1694f700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:10.881239+0000 front 2019-12-11T21:39:10.945466+0000 (oldest deadline 2019-12-11T21:39:56.764389+0000) 2019-12-11T21:39:57.948 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:39:57.940+0000 1694f700 -1 osd.2 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:16.069472+0000 front 2019-12-11T21:39:15.150238+0000 (oldest deadline 2019-12-11T21:39:55.978580+0000) 2019-12-11T21:39:58.274 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:39:58.269+0000 1694f700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.167808+0000 front 2019-12-11T21:39:13.181448+0000 (oldest deadline 2019-12-11T21:39:56.638543+0000) 2019-12-11T21:39:58.539 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:39:58.533+0000 1694f700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:10.881239+0000 front 2019-12-11T21:39:10.945466+0000 (oldest deadline 2019-12-11T21:39:56.764389+0000) 2019-12-11T21:39:58.918 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:39:58.912+0000 1694f700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:12.735757+0000 front 2019-12-11T21:39:13.055189+0000 (oldest deadline 2019-12-11T21:39:58.484176+0000) 2019-12-11T21:39:58.984 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:39:58.978+0000 1694f700 -1 osd.2 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:16.069472+0000 front 2019-12-11T21:39:15.150238+0000 (oldest deadline 2019-12-11T21:39:55.978580+0000) 2019-12-11T21:39:59.022 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:39:59.017+0000 1694f700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.048728+0000 front 2019-12-11T21:39:13.051092+0000 (oldest deadline 2019-12-11T21:39:58.461905+0000) 2019-12-11T21:39:59.095 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:39:59.090+0000 1694f700 -1 osd.0 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:14.259022+0000 front 2019-12-11T21:39:14.300324+0000 (oldest deadline 2019-12-11T21:39:58.351432+0000) 2019-12-11T21:39:59.298 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:39:59.292+0000 1694f700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.167808+0000 front 2019-12-11T21:39:13.181448+0000 (oldest deadline 2019-12-11T21:39:56.638543+0000) 2019-12-11T21:39:59.299 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:39:59.294+0000 1694f700 -1 osd.3 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104053 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5416455408 gen 1] snapc 0=[] ondisk+write+known_if_redirected e152) 2019-12-11T21:39:59.557 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:39:59.549+0000 1694f700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:10.881239+0000 front 2019-12-11T21:39:10.945466+0000 (oldest deadline 2019-12-11T21:39:56.764389+0000) 2019-12-11T21:39:59.917 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:39:59.911+0000 1694f700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:12.735757+0000 front 2019-12-11T21:39:13.055189+0000 (oldest deadline 2019-12-11T21:39:58.484176+0000) 2019-12-11T21:40:00.022 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:40:00.013+0000 1694f700 -1 osd.2 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:16.069472+0000 front 2019-12-11T21:39:15.150238+0000 (oldest deadline 2019-12-11T21:39:55.978580+0000) 2019-12-11T21:40:00.035 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:40:00.029+0000 1694f700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.048728+0000 front 2019-12-11T21:39:13.051092+0000 (oldest deadline 2019-12-11T21:39:58.461905+0000) 2019-12-11T21:40:00.122 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:40:00.117+0000 1694f700 -1 osd.0 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:14.259022+0000 front 2019-12-11T21:39:14.300324+0000 (oldest deadline 2019-12-11T21:39:58.351432+0000) 2019-12-11T21:40:00.264 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:40:00.259+0000 1694f700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.167808+0000 front 2019-12-11T21:39:13.181448+0000 (oldest deadline 2019-12-11T21:39:56.638543+0000) 2019-12-11T21:40:00.264 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:40:00.260+0000 1694f700 -1 osd.3 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104053 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5416455408 gen 1] snapc 0=[] ondisk+write+known_if_redirected e152) 2019-12-11T21:40:00.540 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:40:00.533+0000 1694f700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:10.881239+0000 front 2019-12-11T21:39:10.945466+0000 (oldest deadline 2019-12-11T21:39:56.764389+0000) 2019-12-11T21:40:00.953 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:40:00.947+0000 1694f700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:12.735757+0000 front 2019-12-11T21:39:13.055189+0000 (oldest deadline 2019-12-11T21:39:58.484176+0000) 2019-12-11T21:40:00.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:40:00.978+0000 1694f700 -1 osd.2 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:16.069472+0000 front 2019-12-11T21:39:15.150238+0000 (oldest deadline 2019-12-11T21:39:55.978580+0000) 2019-12-11T21:40:01.054 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:40:01.047+0000 1694f700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.048728+0000 front 2019-12-11T21:39:13.051092+0000 (oldest deadline 2019-12-11T21:39:58.461905+0000) 2019-12-11T21:40:01.130 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:40:01.117+0000 1694f700 -1 osd.0 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:14.259022+0000 front 2019-12-11T21:39:14.300324+0000 (oldest deadline 2019-12-11T21:39:58.351432+0000) 2019-12-11T21:40:01.227 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:40:01.221+0000 1694f700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:39:13.167808+0000 front 2019-12-11T21:39:13.181448+0000 (oldest deadline 2019-12-11T21:39:56.638543+0000) 2019-12-11T21:40:01.227 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:40:01.223+0000 1694f700 -1 osd.3 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104053 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5416455408 gen 1] snapc 0=[] ondisk+write+known_if_redirected e152) 2019-12-11T21:40:02.429 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:40:02.387+0000 1694f700 -1 osd.3 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104053 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5416455408 gen 1] snapc 0=[] ondisk+write+known_if_redirected e152) 2019-12-11T21:40:03.556 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T21:40:03.539+0000 20e25700 -1 RGWWatcher::handle_error cookie 5416455408 err (107) Transport endpoint is not connected 2019-12-11T21:40:08.670 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:40:08.637+0000 31f87700 -1 osd.0 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:08.942 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:40:08.886+0000 31f87700 -1 osd.2 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:08.944 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:40:08.894+0000 31f87700 -1 osd.2 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:08.998 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:40:08.656+0000 31f87700 -1 osd.0 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:09.090 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:40:09.079+0000 33f8b700 -1 osd.0 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:09.118 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:40:09.094+0000 3478c700 -1 osd.2 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:09.217 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:40:09.162+0000 37792700 -1 osd.0 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:09.247 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:40:09.184+0000 37f93700 -1 osd.2 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:40:09.309 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:40:09.254+0000 3478c700 -1 osd.0 156 build_incremental_map_msg missing incremental map 156 2019-12-11T21:41:38.426 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:41:37.977+0000 1694f700 -1 osd.5 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:103918 7.0 7:0b8649ba:::pubsub.user.foo.client.0.bucket.907e8f85919a476e8248d20411406155%2f14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.667:head [getxattrs,stat] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected e159) 2019-12-11T21:42:30.428 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:30.422+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:30.432 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:30.425+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:31.210 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:31.203+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:31.411 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:31.406+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:31.466 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:31.460+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:32.183 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:32.176+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:32.385 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:32.380+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:32.499 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:32.493+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:32.782 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:32.776+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:33.147 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:33.137+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:33.441 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:33.435+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:33.476 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:33.471+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:33.742 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:33.736+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:33.875 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:33.869+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:34.107 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:34.101+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:34.455 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:34.449+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:34.458 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:34.450+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:34.718 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:34.712+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:34.904 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:34.898+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:35.099 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:35.089+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:35.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:35.421+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:35.460 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:35.454+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:35.489 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:35.483+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:35.583 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:35.577+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:35.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:35.679+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:35.925 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:35.919+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:36.143 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:36.135+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:36.439 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:36.433+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:36.439 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:36.431+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:36.538 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:36.532+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:36.614 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:36.608+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:36.703 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:36.698+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:36.880 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:36.874+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:37.138 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:37.132+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:37.405 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:37.398+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:37.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:37.421+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:37.514 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:37.507+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:37.657 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:37.651+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:37.755 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:37.750+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:37.931 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:37.924+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:38.164 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:38.158+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:38.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:38.421+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:38.464 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:38.446+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:38.497 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:38.492+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:38.677 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:38.671+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:38.721 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:38.716+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:38.980 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:38.974+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:39.211 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:39.204+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:39.453 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:39.448+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:39.456 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:39.450+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:39.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:39.468+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:39.679 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:39.673+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:39.763 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:39.757+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:39.841 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:10:26.772 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:39.844 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:10:26.775 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:39.889 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:10:26.790 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:39.901 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:10:26.827 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:39.901 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:10:26.798 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:39.904 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:10:26.834 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:39.963 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:39.957+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:40.082 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:10:27.013 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:40.137 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:10:27.068 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:40.170 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:10:27.071 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:40.230 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:40.224+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:40.471 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:40.465+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:40.488 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:40.483+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:40.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:40.502+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:40.719 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:40.713+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:40.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:40.716+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:40.975 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:40.969+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:41.226 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:41.220+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:41.426 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:41.419+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:41.492 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:41.486+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:41.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:41.500+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:41.726 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:41.720+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:41.745 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:41.738+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:41.953 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:41.946+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:42.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:42.199+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:42.437 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:42.432+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:42.452 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:42.447+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:42.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:42.556+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:42.689 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:42.681+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:42.777 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:42.771+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:42.935 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:42.929+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:43.207 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:43.202+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:43.408 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:43.403+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:43.416 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:43.411+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:43.594 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:43.586+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:43.672 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:43.666+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:43.748 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:43.742+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:43.982 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:43.977+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:44.175 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:44.170+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:44.448 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:44.440+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:44.456 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:44.452+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:44.626 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:44.620+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:44.658 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:44.652+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:44.721 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:44.715+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:45.002 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:44.997+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:45.223 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:45.218+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:45.457 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:45.450+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:45.498 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:45.494+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:45.609 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:45.603+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:45.662 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:45.656+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:45.765 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:45.761+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:46.055 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:46.050+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:46.220 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:46.214+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:46.444 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:46.438+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:46.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:46.504+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:46.587 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:46.580+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:46.683 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:46.676+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:46.734 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:46.728+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:47.075 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:47.070+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:47.173 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:47.168+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:47.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:47.470+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:47.519 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:47.514+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:47.545 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:47.539+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:47.647 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:47.641+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:47.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:47.690+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:48.084 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:48.078+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:48.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:48.159+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:48.510 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:48.505+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:48.512 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:48.506+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:48.516 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:48.510+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:48.637 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:48.632+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:48.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:48.679+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:49.137 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:49.131+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:49.156 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:49.151+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:49.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:49.469+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:49.496 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:49.490+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:49.512 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:49.507+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:49.650 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:49.644+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:49.663 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:49.656+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:49.768 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:10:36.698 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.775 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:10:36.706 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.812 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:10:36.742 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.852 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:10:36.754 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.864 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:10:36.766 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.877 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:10:36.807 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.879 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:10:36.810 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.896 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:10:36.798 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:49.896 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:10:36.827 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:42:50.149 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:50.144+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:50.171 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:50.167+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:50.432 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:50.428+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:50.466 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:50.461+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:50.549 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:50.543+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:50.656 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:50.650+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:50.686 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:50.680+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:51.126 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:51.121+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:51.164 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:51.159+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:51.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:51.424+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:51.480 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:51.475+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:51.552 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:51.545+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:51.634 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:51.629+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:51.642 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:51.636+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:52.129 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:52.119+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:52.214 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:52.208+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:52.422 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:52.416+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:52.485 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:52.479+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:52.557 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:52.551+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:52.632 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:52.627+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:52.657 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:52.643+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:53.119 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:53.114+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:53.247 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:53.242+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:53.392 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:53.386+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:53.443 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:53.438+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:53.582 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:53.576+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:53.670 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:53.659+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:53.670 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:53.662+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:42:54.157 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:42:54.152+0000 1694f700 -1 osd.0 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:49.621445+0000 front 2019-12-11T21:41:49.459600+0000 (oldest deadline 2019-12-11T21:42:33.619821+0000) 2019-12-11T21:42:54.299 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:42:54.294+0000 1694f700 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.897403+0000 front 2019-12-11T21:41:50.024717+0000 (oldest deadline 2019-12-11T21:42:30.747473+0000) 2019-12-11T21:42:54.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:42:54.438+0000 1694f700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.024502+0000 front 2019-12-11T21:41:50.310580+0000 (oldest deadline 2019-12-11T21:42:30.278572+0000) 2019-12-11T21:42:54.465 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:42:54.460+0000 1694f700 -1 osd.1 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.705152+0000 front 2019-12-11T21:41:50.702801+0000 (oldest deadline 2019-12-11T21:42:34.740927+0000) 2019-12-11T21:42:54.564 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:42:54.559+0000 1694f700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.028564+0000 front 2019-12-11T21:41:50.717912+0000 (oldest deadline 2019-12-11T21:42:30.111869+0000) 2019-12-11T21:42:54.670 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:42:54.664+0000 1694f700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:50.274343+0000 front 2019-12-11T21:41:49.649518+0000 (oldest deadline 2019-12-11T21:42:32.722226+0000) 2019-12-11T21:42:54.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:42:54.716+0000 1694f700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:41:53.258968+0000 front 2019-12-11T21:41:53.259533+0000 (oldest deadline 2019-12-11T21:42:35.500777+0000) 2019-12-11T21:43:46.498 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:46.484+0000 31f87700 -1 osd.6 164 build_incremental_map_msg missing incremental map 164 2019-12-11T21:43:46.548 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:46.491+0000 31f87700 -1 osd.6 164 build_incremental_map_msg missing incremental map 164 2019-12-11T21:43:46.565 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:46.538+0000 34f8d700 -1 osd.6 164 build_incremental_map_msg missing incremental map 164 2019-12-11T21:43:46.983 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:46.805+0000 33f8b700 -1 osd.6 164 build_incremental_map_msg missing incremental map 164 2019-12-11T21:43:47.068 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:46.920+0000 3478c700 -1 osd.6 164 build_incremental_map_msg missing incremental map 164 2019-12-11T21:43:52.452 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:52.448+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:53.501 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:53.497+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:53.972 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:43:53.966+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:43:54.147 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:54.140+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:43:54.391 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:43:54.384+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:43:54.511 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:54.507+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:54.984 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:43:54.977+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:43:55.157 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:55.150+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:43:55.373 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:43:55.363+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:43:55.519 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:55.512+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:43:55.522 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:55.518+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:55.968 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:43:55.962+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:43:56.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:43:56.131+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:43:56.179 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:56.172+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:43:56.341 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:43:56.332+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:43:56.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:56.567+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:43:56.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:56.570+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:56.743 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:43:56.729+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:43:57.006 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:43:57.001+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:43:57.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:43:57.146+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:43:57.181 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:57.173+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:43:57.382 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:43:57.376+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:43:57.547 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:57.543+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:43:57.549 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:57.546+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:57.696 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:43:57.690+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:43:58.033 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:43:58.027+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:43:58.134 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:43:58.125+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:43:58.187 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:58.179+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:43:58.387 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:43:58.380+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:43:58.534 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:58.527+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:43:58.535 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:58.530+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:58.716 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:43:58.710+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:43:59.064 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:43:59.058+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:43:59.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:43:59.099+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:43:59.172 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:43:59.164+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:43:59.357 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:43:59.349+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:43:59.513 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:59.508+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:43:59.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:43:59.511+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:43:59.727 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:43:59.721+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:00.103 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:00.098+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:00.110 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:00.102+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:00.185 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:00.177+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:00.401 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:00.395+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:00.527 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:00.522+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:00.529 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:00.525+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:00.699 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:00.694+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:01.065 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:01.058+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:01.138 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:01.133+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:01.170 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:01.164+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:01.430 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:01.424+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:01.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:01.509+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:01.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:01.511+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:01.667 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:01.659+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:02.077 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:02.068+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:02.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:02.088+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:02.191 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:02.183+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:02.414 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:02.408+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:02.473 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:02.466+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:02.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:02.468+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:02.689 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:02.682+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:03.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:03.064+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:03.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:03.123+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:03.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:03.162+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:03.451 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:03.444+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:03.464 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:03.459+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:03.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:03.461+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:03.651 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:03.644+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:04.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:04.063+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:04.095 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:04.088+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:04.213 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:04.197+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:04.412 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:04.406+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:04.478 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:04.471+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:04.479 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:04.472+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:04.628 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:04.619+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:05.124 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:05.118+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:05.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:05.135+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:05.256 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:05.250+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:05.431 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:05.425+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:05.479 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:05.474+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:05.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:05.475+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:05.602 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:05.596+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:06.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:06.110+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:06.161 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:06.155+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:06.254 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:06.247+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:06.483 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:06.465+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:06.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:06.494+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:06.502 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:06.497+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:06.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:06.591+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:07.134 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:07.129+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:07.177 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:07.171+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:07.272 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:07.265+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:07.476 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:07.469+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:07.495 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:07.488+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:07.495 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:07.490+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:07.650 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:07.645+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:08.175 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:08.168+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:08.194 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:08.187+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:08.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:08.239+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:08.493 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:08.486+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:08.508 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:08.500+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:08.508 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:08.503+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:08.662 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:08.655+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:09.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:09.131+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:09.169 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:09.158+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:09.265 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:44:09.258+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.042073+0000 front 2019-12-11T21:43:09.057403+0000 (oldest deadline 2019-12-11T21:43:53.718845+0000) 2019-12-11T21:44:09.493 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:09.489+0000 1694f700 -1 osd.3 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:08.707691+0000 front 2019-12-11T21:43:08.714186+0000 (oldest deadline 2019-12-11T21:43:54.554570+0000) 2019-12-11T21:44:09.494 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:09.490+0000 1694f700 -1 osd.3 166 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:09.533 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:44:09.517+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:09.037552+0000 front 2019-12-11T21:43:09.034700+0000 (oldest deadline 2019-12-11T21:43:54.320607+0000) 2019-12-11T21:44:09.625 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:44:09.618+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.900687+0000 front 2019-12-11T21:43:10.831661+0000 (oldest deadline 2019-12-11T21:43:55.933507+0000) 2019-12-11T21:44:10.162 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:44:10.153+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.957913+0000 front 2019-12-11T21:43:10.951135+0000 (oldest deadline 2019-12-11T21:43:53.836055+0000) 2019-12-11T21:44:10.188 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:44:10.180+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:43:10.515591+0000 front 2019-12-11T21:43:10.375529+0000 (oldest deadline 2019-12-11T21:43:55.646996+0000) 2019-12-11T21:44:10.787 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:44:10.650+0000 1694f700 -1 osd.3 167 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104438 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5458960928 gen 1] snapc 0=[] ondisk+write+known_if_redirected e162) 2019-12-11T21:44:12.369 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T21:44:12.275+0000 20e25700 -1 RGWWatcher::handle_error cookie 5458960928 err (107) Transport endpoint is not connected 2019-12-11T21:44:16.663 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:44:16.658+0000 1694f700 -1 osd.5 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104501 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 41] snapc 0=[] ondisk+write+known_if_redirected e164) 2019-12-11T21:44:17.653 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:44:17.648+0000 1694f700 -1 osd.5 168 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104501 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 41] snapc 0=[] ondisk+write+known_if_redirected e164) 2019-12-11T21:45:09.395 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:45:09.386+0000 3578e700 -1 osd.1 172 build_incremental_map_msg missing incremental map 172 2019-12-11T21:45:09.541 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:45:09.433+0000 34f8d700 -1 osd.1 172 build_incremental_map_msg missing incremental map 172 2019-12-11T21:45:09.549 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:45:09.463+0000 35f8f700 -1 osd.1 172 build_incremental_map_msg missing incremental map 172 2019-12-11T21:45:09.628 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:45:09.471+0000 35f8f700 -1 osd.1 172 build_incremental_map_msg missing incremental map 172 2019-12-11T21:45:09.658 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:45:09.525+0000 35f8f700 -1 osd.1 172 build_incremental_map_msg missing incremental map 172 2019-12-11T21:45:09.751 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:45:09.616+0000 3378a700 -1 osd.1 172 build_incremental_map_msg missing incremental map 172 2019-12-11T21:46:05.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:05.548+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:05.752 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:05.744+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:06.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:06.380+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:06.527 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:06.521+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:06.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:06.737+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:07.226 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:07.221+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:07.396 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:07.391+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:07.533 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:07.528+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:07.729 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:07.712+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:08.278 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:08.272+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:08.443 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:08.438+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:08.546 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:08.540+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:08.592 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:08.586+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:08.685 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:08.680+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:08.782 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:08.775+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:09.319 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:09.314+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:09.483 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:09.478+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:09.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:09.575+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:09.608 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:09.602+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:09.673 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:09.666+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:09.830 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:09.824+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:10.333 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:10.327+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:10.360 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:10.354+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:10.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:10.469+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:10.601 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:10.596+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:10.602 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:10.597+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:10.630 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:10.626+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:10.860 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:10.855+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:11.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:11.311+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:11.328 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:11.324+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:11.451 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:11.446+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:11.562 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:11.557+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:11.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:11.598+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:11.650 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:11.646+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:11.838 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:11.833+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:12.305 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:12.299+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:12.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:12.342+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:12.452 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:12.447+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:12.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:12.512+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:12.559 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:12.554+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:12.666 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:12.662+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:12.866 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:12.861+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:13.351 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:13.347+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:13.358 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:13.353+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:13.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:13.469+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:13.526 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:13.521+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:13.542 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:13.537+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:13.629 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:13.623+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:13.850 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:13.846+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:14.379 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:14.372+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:14.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:14.382+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:14.457 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:14.451+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:14.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:14.510+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:14.541 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:14.534+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:14.616 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:14.611+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:14.825 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:14.820+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:15.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:15.414+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:15.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:15.423+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:15.498 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:15.492+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:15.502 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:15.496+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:15.568 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:15.561+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:15.644 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:15.638+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:15.789 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:15.784+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:16.473 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:16.467+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:16.474 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:16.468+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:16.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:16.468+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:16.517 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:16.511+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:16.543 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:16.537+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:16.613 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:16.608+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:16.829 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:16.824+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:17.443 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:17.438+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:17.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:17.442+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:17.484 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:17.477+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:17.489 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:17.479+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:17.572 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:17.566+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:17.601 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:17.596+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:17.836 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:17.830+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:18.431 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:18.425+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:18.449 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:18.441+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:18.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:18.511+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:18.530 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:18.524+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:18.576 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:18.571+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:18.599 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:18.593+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:18.887 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:18.880+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:19.405 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:19.399+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:19.437 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:19.430+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:19.508 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:19.501+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:19.553 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:19.547+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:19.561 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:19.555+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:19.576 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:19.570+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:19.868 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:19.862+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:20.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:20.364+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:20.418 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:20.413+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:20.507 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:20.501+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:20.575 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:20.571+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:20.575 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:20.569+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:20.585 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:20.580+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:20.892 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:20.885+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:21.404 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:21.399+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:21.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:21.414+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:21.470 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:21.465+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:21.535 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:21.530+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:21.552 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:21.547+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:21.627 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:21.623+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:21.900 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:21.894+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:22.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:22.434+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:22.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:22.438+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:22.447 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:22.442+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:22.544 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:22.539+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:22.588 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:22.582+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:22.629 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:22.623+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:22.871 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:22.866+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:23.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:23.405+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:23.430 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:23.424+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:23.466 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:23.460+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:23.571 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:23.565+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:23.575 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:23.569+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:23.613 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:23.608+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:23.877 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:23.872+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:24.418 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:24.412+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:24.423 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:24.417+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:24.470 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:24.462+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:24.611 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:24.606+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:24.612 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:24.606+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:24.653 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:24.648+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:24.873 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:24.867+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:25.418 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:25.412+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:25.429 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:25.422+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:25.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:25.428+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:25.600 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:25.593+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:25.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:25.598+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:25.697 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:25.692+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:25.910 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:25.905+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:26.392 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:26.387+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:26.425 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:26.410+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:26.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:26.464+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:26.583 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:26.577+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:26.591 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:26.584+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:26.755 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:26.741+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:26.887 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:26.876+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:27.376 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:27.370+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:27.383 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:27.377+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:27.453 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:27.441+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:27.555 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:27.549+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:27.573 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:27.567+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:27.718 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:27.713+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:27.876 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:27.870+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:28.347 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:28.340+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:28.387 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:28.382+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:28.481 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:28.476+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:28.586 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:28.581+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:28.617 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:28.611+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:28.689 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:28.682+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:28.909 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:28.902+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:29.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:29.380+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:29.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:29.402+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:29.462 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:29.453+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:29.540 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:29.536+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:29.618 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:29.612+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:29.706 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:29.701+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:29.892 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:29.886+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:30.406 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:30.401+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:30.436 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:30.431+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:30.469 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:30.462+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:30.574 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:30.568+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:30.597 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:30.584+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:30.721 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:30.716+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:30.877 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:30.871+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:31.441 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:31.435+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:31.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:31.461+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:31.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:31.500+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:31.560 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:31.555+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:31.598 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:31.593+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:31.758 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:31.752+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:31.928 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:31.921+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:32.436 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:32.430+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:32.493 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:32.487+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:32.495 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:32.490+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:32.598 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:32.592+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:32.610 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:32.605+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:32.725 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:32.720+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:32.913 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:32.906+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:33.425 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:33.419+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:33.492 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:33.472+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:33.529 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:33.523+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:33.586 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:33.579+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:33.620 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:33.615+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:33.692 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:33.687+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:33.931 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:33.925+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:34.412 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:34.405+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:34.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:34.429+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:34.548 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:34.542+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:34.555 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:34.549+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:34.624 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:34.618+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:34.737 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:34.732+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:34.951 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:46:34.945+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:22.586534+0000 front 2019-12-11T21:45:25.328438+0000 (oldest deadline 2019-12-11T21:46:05.105491+0000) 2019-12-11T21:46:35.428 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:46:35.422+0000 1694f700 -1 osd.1 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:19.936718+0000 front 2019-12-11T21:45:25.922896+0000 (oldest deadline 2019-12-11T21:46:05.795925+0000) 2019-12-11T21:46:35.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:46:35.468+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:25.612038+0000 front 2019-12-11T21:45:24.659468+0000 (oldest deadline 2019-12-11T21:46:10.311060+0000) 2019-12-11T21:46:35.562 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:46:35.557+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:23.671895+0000 front 2019-12-11T21:45:23.810174+0000 (oldest deadline 2019-12-11T21:46:06.928244+0000) 2019-12-11T21:46:35.606 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:46:35.600+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.491781+0000 front 2019-12-11T21:45:24.561243+0000 (oldest deadline 2019-12-11T21:46:08.478484+0000) 2019-12-11T21:46:35.661 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:46:35.653+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.399182+0000 front 2019-12-11T21:45:25.633301+0000 (oldest deadline 2019-12-11T21:46:08.152179+0000) 2019-12-11T21:46:35.755 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:46:35.751+0000 1694f700 -1 osd.3 172 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:45:24.029551+0000 front 2019-12-11T21:45:25.792247+0000 (oldest deadline 2019-12-11T21:46:05.359515+0000) 2019-12-11T21:46:38.007 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:14:24.934 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.021 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:14:24.942 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.021 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:14:24.920 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.024 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:14:24.925 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.025 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:14:24.956 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.032 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:14:24.960 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.250 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:14:25.151 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.308 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:14:25.238 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:46:38.353 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:14:25.284 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:47:18.872 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:18.842+0000 3578e700 -1 osd.0 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:18.920 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:18.848+0000 3578e700 -1 osd.0 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.120 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:18.917+0000 35f8f700 -1 osd.0 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.211 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:19.202+0000 35f8f700 -1 osd.2 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.216 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:18.988+0000 35f8f700 -1 osd.0 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.238 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:19.208+0000 35f8f700 -1 osd.2 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.274 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:19.230+0000 31f87700 -1 osd.2 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.381 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:19.275+0000 3478c700 -1 osd.0 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:19.411+0000 33f8b700 -1 osd.2 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:19.672 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:19.621+0000 3478c700 -1 osd.2 176 build_incremental_map_msg missing incremental map 176 2019-12-11T21:47:21.115 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:21.069+0000 3578e700 -1 osd.5 177 build_incremental_map_msg missing incremental map 177 2019-12-11T21:47:22.456 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:22.418+0000 38794700 -1 osd.5 178 build_incremental_map_msg missing incremental map 177 2019-12-11T21:47:32.505 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:32.497+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:33.489 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:33.484+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:34.055 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:34.047+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:34.464 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:34.460+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:34.705 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:34.697+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:35.104 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:35.096+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:35.443 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:35.438+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:35.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:35.717+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:36.095 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:36.088+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:36.175 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:36.171+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:36.411 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:36.405+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:36.411 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:36.407+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:36.768 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:36.756+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:37.090 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:37.084+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:37.169 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:37.158+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:37.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:37.164+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:37.445 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:37.441+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:37.446 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:37.443+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:37.729 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:37.724+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:38.089 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:38.082+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:38.158 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:38.153+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:38.184 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:38.177+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:38.458 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:38.452+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:38.458 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:38.454+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:38.754 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:38.748+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:38.906 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:38.899+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:39.045 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:39.038+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:39.148 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:39.141+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:39.169 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:39.162+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:39.419 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:39.413+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:39.420 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:39.415+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:39.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:39.783+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:39.868 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:39.861+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:40.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:39.998+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:40.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:40.143+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:40.177 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:40.172+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:40.380 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:40.371+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:40.380 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:40.373+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:40.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:40.782+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:40.885 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:40.881+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:41.013 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:41.005+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:41.191 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:41.184+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:41.217 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:41.209+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:41.429 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:41.418+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:41.430 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:41.422+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:41.747 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:41.739+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:41.915 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:41.909+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:41.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:41.963+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:42.199 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:42.192+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:42.209 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:42.202+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:42.384 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:42.378+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:42.387 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:42.382+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:42.794 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:42.787+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:42.910 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:42.903+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:42.966 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:42.959+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:43.203 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:43.195+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:43.204 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:43.196+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:43.359 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:43.352+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:43.361 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:43.356+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:43.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:43.765+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:43.902 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:43.897+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:44.015 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:44.007+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:44.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:44.158+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:44.246 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:44.238+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:44.358 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:44.352+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:44.358 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:44.353+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:44.754 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:44.747+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:44.916 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:44.909+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:45.051 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:45.044+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:45.204 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:45.197+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:45.236 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:45.230+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:45.382 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:45.377+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:45.383 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:45.378+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:45.763 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:45.756+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:45.880 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:45.873+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:46.081 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:46.068+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:46.198 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:46.192+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:46.243 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:46.236+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:46.419 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:46.411+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:46.419 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:46.413+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:46.759 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:46.752+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:46.908 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:46.901+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:47.136 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:47.126+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:47.189 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:47.183+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:47.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:47.205+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:47.387 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:47.379+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:47.388 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:47.384+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:47.772 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:47.764+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:47.920 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:47.915+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:48.153 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:48.145+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:48.202 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:48.195+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:48.203 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:48.199+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:48.351 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:48.343+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:48.352 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:48.346+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:48.814 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:48.807+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:48.926 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:48.917+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:49.144 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:49.138+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:49.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:49.151+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:49.234 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:49.228+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:49.376 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:49.371+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:49.378 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:49.374+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:49.861 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:49.854+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:49.882 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:49.877+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:50.148 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:50.140+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:50.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:50.167+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:50.231 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:50.226+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:50.412 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:50.407+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:50.412 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:50.408+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:50.821 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:50.815+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:50.860 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:50.851+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:51.069 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:51.063+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:51.138 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:51.131+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:51.196 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:51.190+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:51.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:51.270+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:51.464 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:51.459+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:51.464 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:51.460+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:51.818 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:51.811+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:51.828 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:51.824+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:52.119 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:52.114+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:52.128 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:52.121+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:52.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:52.158+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:52.313 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:52.308+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:52.478 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:52.473+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:52.479 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:52.474+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:52.789 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:52.777+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:52.815 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:52.809+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:53.129 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:53.123+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:53.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:53.157+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:53.171 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:53.164+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:53.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:53.290+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:53.444 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:53.437+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:53.444 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:53.439+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:53.765 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:53.758+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:53.824 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:53.819+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:54.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:54.119+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:54.141 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:54.135+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:54.177 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:54.170+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:54.254 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:54.247+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:54.473 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:54.467+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:54.473 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:54.469+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:54.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:54.736+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:54.823 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:54.817+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:55.123 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:55.117+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:55.139 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:55.132+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:55.220 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:55.216+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:55.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:55.221+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:55.438 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:55.433+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:55.439 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:55.434+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:55.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:55.784+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:55.818 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:55.814+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:56.103 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:56.097+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:56.166 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:56.161+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:56.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:56.179+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:56.267 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:56.262+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:56.471 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:56.466+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:56.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:56.470+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:56.823 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:56.819+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:56.829 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:56.823+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:57.136 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:57.131+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:57.150 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:57.140+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:47:16.866391+0000 (oldest deadline 2019-12-11T21:47:56.866391+0000) 2019-12-11T21:47:57.151 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:57.145+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:57.166 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:57.160+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:57.298 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:57.292+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:57.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:57.501+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:57.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:57.503+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:57.835 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:57.830+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:57.845 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:57.838+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:58.117 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:58.110+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:58.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:58.145+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:58.164 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:58.156+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:47:16.866391+0000 (oldest deadline 2019-12-11T21:47:56.866391+0000) 2019-12-11T21:47:58.165 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:58.160+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:58.301 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:58.295+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:58.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:58.503+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:58.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:58.505+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:58.790 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:58.785+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:58.892 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:58.884+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:47:59.089 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:47:59.081+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:47:59.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:47:59.146+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:47:59.175 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:59.167+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:47:16.866391+0000 (oldest deadline 2019-12-11T21:47:56.866391+0000) 2019-12-11T21:47:59.176 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:47:59.170+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:47:59.307 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:47:59.302+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:47:59.497 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:59.490+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:47:59.498 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:47:59.493+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:47:59.766 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:47:59.761+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:47:59.931 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:47:59.924+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:48:00.129 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:48:00.121+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.624797+0000 front 2019-12-11T21:46:52.608139+0000 (oldest deadline 2019-12-11T21:47:36.660058+0000) 2019-12-11T21:48:00.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:48:00.187+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.122808+0000 front 2019-12-11T21:46:52.059033+0000 (oldest deadline 2019-12-11T21:47:33.716621+0000) 2019-12-11T21:48:00.216 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:48:00.210+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:47:16.866391+0000 (oldest deadline 2019-12-11T21:47:56.866391+0000) 2019-12-11T21:48:00.219 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:48:00.214+0000 1694f700 -1 osd.5 178 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:48:00.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:48:00.354+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:51.408829+0000 front 2019-12-11T21:46:51.247293+0000 (oldest deadline 2019-12-11T21:47:35.330158+0000) 2019-12-11T21:48:00.510 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:48:00.504+0000 1694f700 -1 osd.3 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.559815+0000 front 2019-12-11T21:46:52.566344+0000 (oldest deadline 2019-12-11T21:47:35.442323+0000) 2019-12-11T21:48:00.513 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:48:00.509+0000 1694f700 -1 osd.3 178 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:48:00.737 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:48:00.730+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:52.783251+0000 front 2019-12-11T21:46:52.783789+0000 (oldest deadline 2019-12-11T21:47:38.655894+0000) 2019-12-11T21:48:00.940 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:48:00.932+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:46:50.020488+0000 front 2019-12-11T21:46:50.078121+0000 (oldest deadline 2019-12-11T21:47:34.104367+0000) 2019-12-11T21:48:01.419 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:48:01.327+0000 1694f700 -1 osd.5 179 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e176) 2019-12-11T21:48:01.508 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:48:01.331+0000 34f8d700 -1 osd.5 179 build_incremental_map_msg missing incremental map 177 2019-12-11T21:48:01.815 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:48:01.538+0000 1694f700 -1 osd.3 179 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:48:02.752 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:48:02.543+0000 1694f700 -1 osd.3 180 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:104822 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 821156640 gen 1] snapc 0=[] ondisk+write+known_if_redirected e174) 2019-12-11T21:48:03.276 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T21:48:03.254+0000 20e25700 -1 RGWWatcher::handle_error cookie 821156640 err (107) Transport endpoint is not connected 2019-12-11T21:48:47.712 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:48:47.620+0000 3578e700 -1 osd.1 183 build_incremental_map_msg missing incremental map 183 2019-12-11T21:48:47.730 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:48:47.622+0000 30f85700 -1 osd.1 183 build_incremental_map_msg missing incremental map 183 2019-12-11T21:48:47.773 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:48:47.722+0000 34f8d700 -1 osd.1 183 build_incremental_map_msg missing incremental map 183 2019-12-11T21:48:48.215 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:48:48.188+0000 31786700 -1 osd.1 183 build_incremental_map_msg missing incremental map 183 2019-12-11T21:49:46.913 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:46.907+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:47.928 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:47.923+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:48.055 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:48.049+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:48.355 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:48.348+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:48.595 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:48.589+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:48.981 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:48.975+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:49.019 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:49.013+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:49.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:49.319+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:49.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:49.574+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:50.004 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:49.999+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:50.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:50.034+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:50.271 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:50.265+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:50.317 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:50.311+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:50.600 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:50.594+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:50.877 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:50.871+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:51.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:51.010+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:51.092 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:51.086+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:51.262 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:51.257+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:51.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:51.315+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:51.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:51.548+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:51.570 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:51.565+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:51.919 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:51.913+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:52.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:52.010+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:52.072 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:52.065+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:52.303 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:52.296+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:52.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:52.365+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:52.546 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:52.541+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:52.601 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:52.596+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:52.952 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:52.947+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:53.005 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:52.997+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:53.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:53.033+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:53.334 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:53.329+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:53.415 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:53.409+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:53.526 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:53.519+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:53.571 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:53.566+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:53.964 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:53.958+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:53.983 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:53.976+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:54.029 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:54.023+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:54.345 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:54.338+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:54.442 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:54.436+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:54.490 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:54.484+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:54.531 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:54.524+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:54.964 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:54.958+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:54.986 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:54.978+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:55.027 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:55.021+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:55.399 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:55.391+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:55.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:55.434+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:55.482 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:55.475+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:55.521 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:55.514+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:55.950 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:55.943+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:55.979 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:55.973+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:55.979 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:55.972+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:56.394 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:56.387+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:56.412 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:56.406+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:56.505 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:56.495+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:56.510 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:56.504+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:56.952 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:56.946+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:56.955 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:56.949+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:56.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:56.971+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:57.359 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:57.353+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:57.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:57.383+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:57.465 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:57.459+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:57.557 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:57.549+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:57.938 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:57.931+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:57.969 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:57.962+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:57.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:57.971+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:58.391 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:58.386+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:58.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:58.405+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:58.426 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:58.416+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:58.551 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:58.544+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:58.926 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:58.920+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:58.976 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:58.971+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:49:59.009 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:49:58.997+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:49:59.358 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:49:59.352+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:49:59.402 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:49:59.397+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:49:59.437 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:49:59.432+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:49:59.550 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:49:59.545+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:49:59.960 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:49:59.954+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:49:59.966 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:49:59.961+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:00.053 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:00.048+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:00.363 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:00.359+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:00.368 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:00.362+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:00.469 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:00.464+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:00.583 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:00.579+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:00.960 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:00.954+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:00.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:00.967+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:01.053 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:01.047+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:01.379 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:01.374+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:01.389 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:01.384+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:01.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:01.439+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:01.609 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:01.603+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:01.944 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:01.939+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:01.958 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:01.952+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:02.099 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:02.094+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:02.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:02.348+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:02.375 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:02.364+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:02.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:02.402+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:02.634 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:02.628+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:02.906 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:02.901+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:02.950 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:02.943+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:03.063 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:03.056+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:03.333 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:03.327+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:03.392 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:03.387+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:03.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:03.398+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:03.590 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:03.585+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:03.874 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:03.869+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:03.921 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:03.915+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:04.108 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:04.102+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:04.354 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:04.349+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:04.358 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:04.353+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:04.418 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:04.412+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:04.602 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:04.597+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:04.836 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:04.830+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:04.920 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:04.902+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:05.128 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:05.123+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:05.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:05.314+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:05.350 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:05.345+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:05.441 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:05.436+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:05.578 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:05.565+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:05.864 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:05.854+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:05.882 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:05.875+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:06.115 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:06.110+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:06.355 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:06.347+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:06.404 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:06.398+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:06.462 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:06.456+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:06.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:06.576+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:06.844 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:06.838+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:07.135 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:06.895+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:07.135 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:07.100+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:07.334 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:07.328+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:07.356 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:07.351+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:07.456 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:07.449+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:07.576 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:07.570+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:07.817 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:07.810+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:07.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:07.931+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:08.117 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:08.112+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:08.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:08.333+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:08.351 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:08.344+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:08.444 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:08.439+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:08.570 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:08.565+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:08.779 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:08.773+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:08.972 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:08.965+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:09.095 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:09.091+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:09.316 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:09.312+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:09.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:09.379+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:09.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:09.444+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:09.586 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:09.582+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:09.758 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:09.753+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:09.929 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:09.924+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:10.113 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:10.108+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:10.293 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:10.287+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:10.341 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:10.335+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:10.412 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:10.406+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:10.629 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:10.624+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:10.769 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:10.759+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:10.903 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:10.898+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:11.160 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:11.156+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:11.303 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:11.298+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:11.359 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:11.354+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:11.370 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:11.365+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:11.605 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:11.600+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:11.754 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:11.748+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:11.900 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:11.895+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:12.128 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:12.123+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:12.284 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:12.278+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:12.355 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:12.350+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:12.395 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:12.387+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:12.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:12.582+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:12.755 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:12.750+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:12.875 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:12.869+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:13.086 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:13.081+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:13.337 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:13.332+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:13.390 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:13.383+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:13.412 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:13.406+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:13.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:13.569+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:13.746 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:13.739+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:13.852 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:13.847+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:14.072 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:14.066+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:14.346 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:14.340+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:14.373 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:14.368+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:14.373 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:14.368+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:14.544 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:14.537+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:14.785 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:14.778+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:14.805 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:14.801+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:15.089 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:15.084+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:15.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:15.311+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:15.342 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:15.337+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:15.397 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:15.390+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:15.565 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:15.559+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:15.754 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:15.748+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:15.818 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:15.813+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:16.058 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:16.051+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:16.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:16.308+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:16.368 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:16.362+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:16.432 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:16.426+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:16.579 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:16.575+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:16.803 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:16.797+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:16.821 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:16.815+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:17.110 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:17.104+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:17.350 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:17.345+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:17.356 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:17.351+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:17.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:17.405+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:17.547 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:17.542+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:17.784 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:17.778+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:17.839 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:17.834+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:18.116 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:18.111+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:18.307 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:18.302+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:18.355 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:18.348+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:18.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:18.397+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:18.557 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:18.552+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:18.787 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:18.775+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:18.798 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:18.794+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:19.148 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:19.143+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:19.271 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:19.265+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:19.327 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:19.321+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:19.383 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:19.377+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:19.528 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:19.523+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:19.761 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:19.755+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:19.801 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:19.796+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:20.110 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:20.101+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:20.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:20.317+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:20.343 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:20.337+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:20.364 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:20.358+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:20.519 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:50:20.513+0000 1694f700 -1 osd.3 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:02.467952+0000 front 2019-12-11T21:49:05.235486+0000 (oldest deadline 2019-12-11T21:49:47.584270+0000) 2019-12-11T21:50:20.763 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:50:20.758+0000 1694f700 -1 osd.1 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:05.235575+0000 front 2019-12-11T21:49:04.728296+0000 (oldest deadline 2019-12-11T21:49:46.869586+0000) 2019-12-11T21:50:20.777 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:20.771+0000 1694f700 -1 osd.4 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.478403+0000 front 2019-12-11T21:49:06.386226+0000 (oldest deadline 2019-12-11T21:49:47.822330+0000) 2019-12-11T21:50:21.155 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:50:21.150+0000 1694f700 -1 osd.0 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.041760+0000 front 2019-12-11T21:49:06.096510+0000 (oldest deadline 2019-12-11T21:49:50.099332+0000) 2019-12-11T21:50:21.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:50:21.329+0000 1694f700 -1 osd.7 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:04.190179+0000 front 2019-12-11T21:49:02.513802+0000 (oldest deadline 2019-12-11T21:49:47.787789+0000) 2019-12-11T21:50:21.344 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:50:21.340+0000 1694f700 -1 osd.2 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:07.099856+0000 front 2019-12-11T21:49:07.114966+0000 (oldest deadline 2019-12-11T21:49:50.641562+0000) 2019-12-11T21:50:21.358 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:50:21.352+0000 1694f700 -1 osd.6 183 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:49:06.179978+0000 front 2019-12-11T21:49:05.809466+0000 (oldest deadline 2019-12-11T21:49:50.152792+0000) 2019-12-11T21:50:21.361 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:50:21.349+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 42sec e183 v183) 2019-12-11T21:50:33.865 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:20.796 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:33.877 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:20.808 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:33.888 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:20.790 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:33.889 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:20.819 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:33.893 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:20.823 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:33.895 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:20.797 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:34.150 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:21.080 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:34.158 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:21.088 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:34.271 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:21.172 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.659 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:23.590 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.662 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:23.593 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.693 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:23.624 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.727 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:23.657 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.728 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:23.659 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.732 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:23.632 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.733 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:23.635 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.744 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:23.674 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:36.751 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:23.653 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:50:52.022 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:50:52.017+0000 1694f700 -1 osd.4 185 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105140 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 46] snapc 0=[] ondisk+write+known_if_redirected e184) 2019-12-11T21:51:11.949 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:58.880 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:11.960 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:58.890 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:11.972 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:58.899 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:12.004 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:58.934 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:12.019 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:18:58.949 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:12.025 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:58.926 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:12.030 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:58.932 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:12.053 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:18:58.983 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:12.098 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:18:59.000 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:51:14.677 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:14.673+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:15.679 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:15.675+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:16.704 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:16.699+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:17.720 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:17.716+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:18.456 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:18.450+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:18.772 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:18.767+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:19.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:19.498+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:19.510 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:19.502+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:19.800 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:19.796+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:20.030 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:20.023+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:20.098 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:20.091+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:20.476 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:20.470+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:20.485 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:20.478+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:20.774 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:20.769+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:20.998 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:20.989+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:21.018 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:21.011+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:21.070 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:21.064+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:21.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:21.426+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:21.521 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:21.515+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:21.731 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:21.726+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:22.014 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:22.007+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:22.036 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:22.030+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:22.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:22.052+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:22.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:22.386+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:22.576 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:22.569+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:22.780 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:22.775+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:22.979 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:22.972+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:23.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:23.059+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:23.098 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:23.090+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:23.369 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:23.362+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:23.598 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:23.589+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:23.739 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:23.734+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:23.939 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:23.933+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:24.054 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:24.046+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:24.104 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:24.099+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:24.357 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:24.350+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:24.562 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:24.554+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:24.741 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:24.736+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:24.946 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:24.938+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:25.057 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:25.049+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:25.087 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:25.080+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:25.348 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:25.341+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:25.599 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:25.591+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:25.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:25.773+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:25.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:25.777+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:25.919 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:25.912+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:26.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:26.051+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:26.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:26.107+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:26.388 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:26.383+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:26.636 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:26.629+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:26.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:26.756+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:26.763 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:26.758+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:26.965 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:26.957+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:27.081 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:27.074+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:27.149 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:27.143+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:27.423 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:27.417+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:27.604 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:27.597+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:27.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:27.799+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:27.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:27.800+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:28.002 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:27.994+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:28.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:28.101+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:28.169 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:28.162+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:28.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:28.394+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:28.565 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:28.558+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:28.812 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:28.805+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:28.813 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:28.807+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:28.964 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:28.956+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:29.131 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:29.125+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:29.219 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:29.214+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:29.356 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:29.351+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:29.595 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:29.587+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:29.800 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:29.795+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:29.801 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:29.798+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:30.013 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:30.006+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:30.161 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:30.155+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:30.184 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:30.178+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:30.189 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:30.185+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:30.310 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:30.305+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:30.557 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:30.551+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:30.769 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:30.764+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:30.770 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:30.765+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:31.043 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:31.035+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:31.156 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:31.150+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:31.189 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:31.184+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:31.202 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:31.195+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:31.362 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:31.357+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:31.556 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:31.549+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:31.776 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:31.771+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:31.778 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:31.774+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:32.015 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:32.008+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:32.178 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:32.173+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:32.182 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:32.176+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:32.238 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:32.233+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:32.338 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:32.332+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:32.538 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:32.532+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:32.796 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:32.790+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:32.798 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:32.792+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:32.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:32.971+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:33.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:33.184+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:33.206 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:33.200+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:33.262 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:33.256+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:33.380 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:33.374+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:33.579 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:33.572+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:33.803 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:33.797+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:33.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:33.799+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:33.939 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:33.933+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:34.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:34.227+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:34.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:34.229+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:34.253 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:34.247+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:34.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:34.329+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:34.625 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:34.617+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:34.840 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:34.833+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:34.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:34.836+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:34.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:34.968+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:35.210 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:35.205+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:35.254 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:35.249+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:35.292 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:35.285+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:35.300 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:35.295+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:35.631 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:35.625+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:35.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:35.837+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:35.844 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:35.840+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:35.967 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:35.960+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:36.191 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:36.184+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:36.194 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:36.189+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:36.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:36.243+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:36.291 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:36.287+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:36.306 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:36.301+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:36.677 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:36.670+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:36.882 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:36.876+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:36.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:36.878+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:36.942 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:36.935+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:37.176 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:37.168+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:37.177 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:37.171+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:37.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:37.217+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:37.288 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:37.282+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:37.357 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:37.351+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:37.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:37.628+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:37.880 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:37.874+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:37.882 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:37.878+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:37.931 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:37.924+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:38.192 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:38.183+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:38.193 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:38.187+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:38.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:38.205+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:38.303 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:38.298+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:38.388 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:38.381+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:38.603 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:38.596+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:38.920 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:38.913+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:38.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:38.924+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:38.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:38.927+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:39.199 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:39.191+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:39.241 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:39.235+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:39.244 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:39.239+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:39.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:39.330+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:39.383 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:39.378+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:39.565 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:39.559+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:39.946 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:39.940+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:39.950 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:39.945+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:39.972 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:39.965+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:40.221 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:40.205+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:40.223 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:40.217+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:40.241 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:40.234+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:40.304 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:40.299+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:40.423 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:40.417+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:40.597 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:40.591+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:40.938 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:40.932+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:40.974 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:40.968+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:40.974 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:40.970+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:41.205 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:41.198+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:41.209 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:41.203+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:41.209 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:41.202+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:41.350 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:41.344+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:41.466 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:41.461+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:41.575 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:41.568+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:41.927 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:41.922+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:41.928 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:41.925+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:41.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:41.970+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:42.175 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:42.167+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:42.242 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:42.233+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:42.243 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:42.237+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:42.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:42.362+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:42.447 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:42.440+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:42.554 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:42.547+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:42.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:42.902+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:42.910 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:42.904+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:43.015 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:43.008+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:43.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:43.168+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:43.218 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:43.210+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:43.219 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:43.213+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:43.383 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:43.377+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:43.486 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:43.476+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:43.568 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:43.561+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:43.901+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:43.903+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:44.009 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:44.001+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:44.202 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:44.194+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:44.211 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:44.203+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:44.212 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:44.206+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:44.425 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:44.418+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:44.461 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:44.455+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:44.618 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:44.611+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:44.919 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:44.911+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:44.919 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:44.915+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:44.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:44.992+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:45.182 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:45.175+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:45.228 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:45.221+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:45.233 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:45.227+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:45.413 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:45.407+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:45.446 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:45.441+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:45.626 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:45.619+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:45.930 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:45.925+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:45.933 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:45.929+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:45.989 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:45.981+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:46.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:46.185+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:46.193 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:46.184+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:46.194 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:46.187+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:46.385 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:46.378+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:46.422 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:46.416+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:46.592 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:46.585+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:46.927 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:46.921+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:46.928 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:46.923+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:46.958 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:46.951+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:47.157 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:47.148+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:47.157 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:47.152+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:47.196 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:47.189+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:47.432 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:47.426+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:47.475 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:47.468+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:47.574 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:47.567+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:47.904 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:47.896+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:47.904 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:47.898+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:48.008 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:48.001+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:48.167 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:48.159+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:48.170 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:48.162+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:48.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:48.167+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:48.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:48.414+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:48.459 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:48.454+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:48.584 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:48.578+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:48.933 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:48.927+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:48.934 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:48.929+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:49.044 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:49.037+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:49.194 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:49.185+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:49.194 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:49.189+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:49.199 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:49.192+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:49.407 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:49.402+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:49.425 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:49.419+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:49.586 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:49.578+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:49.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:49.931+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:49.938 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:49.933+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:50.049 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:50.042+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:50.202 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:50.186+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:50.203 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:50.188+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:50.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:50.231+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:50.371 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:50.366+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:50.475 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:50.470+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:50.572 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:50.566+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:50.983 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:50.975+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:50.983 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:50.979+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:51.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:51.030+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:51.181 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:51.162+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:51.181 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:51.165+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:51.274 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:51.267+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:51.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:51.416+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:51.508 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:51.503+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:51.538 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:51.530+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:51.968 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:51.963+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:51.969 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:51.965+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:52.014 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:51:52.008+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e189 v189) 2019-12-11T21:51:52.091 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:52.085+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:52.192 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:52.186+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:52.193 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:52.188+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:52.291 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:52.285+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:52.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:52.460+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:52.502 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:52.495+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:52.512 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:52.507+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:52.978 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:52.973+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:52.980 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:52.976+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:53.117 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:53.111+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:53.224 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:53.216+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:53.224 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:53.219+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:53.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:53.256+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:53.461 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:53.456+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:53.495 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:53.489+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:53.497 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:53.491+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:53.973 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:53.968+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:53.974 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:53.971+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:54.159 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:54.152+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:54.227 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:54.220+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:54.228 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:54.223+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:54.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:54.308+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:54.474 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:54.469+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:54.508 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:54.501+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:54.523 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:54.518+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:55.015 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:55.009+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:55.016 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:55.011+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:55.185 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:55.178+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:55.264 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:55.257+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:55.265 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:55.260+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:55.340 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:55.334+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:55.449 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:55.442+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:55.481 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:55.474+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:55.502 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:55.495+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:56.050 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:56.045+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:56.052 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:56.048+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:56.218 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:56.212+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:56.230 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:56.223+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:56.231 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:56.226+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:56.300 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:56.294+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:56.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:56.400+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:56.465 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:56.460+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:56.481 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:56.475+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:57.041 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:51:57.035+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e189 v189) 2019-12-11T21:51:57.045 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:57.040+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:57.046 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:57.041+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:57.210 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:57.203+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:57.248 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:57.241+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:57.249 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:57.244+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:57.317 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:57.310+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:57.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:57.361+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:57.462 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:57.456+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:57.499 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:57.493+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:58.072 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:58.067+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:58.075 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:58.071+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:58.249 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:58.238+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:58.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:58.272+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:58.290 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:58.284+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:58.294 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:58.289+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:58.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:58.395+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:58.436 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:58.430+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:58.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:58.473+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:51:59.031 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:59.027+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:51:59.033 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:51:59.029+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:51:59.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:51:59.271+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:51:59.282 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:51:59.275+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:51:59.341 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:59.333+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:51:59.342 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:51:59.336+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:51:59.434 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:51:59.428+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:51:59.442 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:51:59.435+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:51:59.469 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:51:59.463+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:00.009 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:00.004+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:00.010 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:00.007+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:00.239 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:00.232+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:00.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:00.273+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:00.380 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:00.373+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:00.384 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:00.379+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:00.429 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:00.422+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:00.431 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:00.424+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:00.469 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:00.462+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:01.023 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:01.016+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:01.024 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:01.019+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:01.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:01.257+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:01.265 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:01.259+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:01.405 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:01.400+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:01.415 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:01.406+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:01.416 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:01.409+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:01.457 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:01.451+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:01.488 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:01.483+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:02.031 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:02.025+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:02.033 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:02.027+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:02.049 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:52:02.046+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e189 v189) 2019-12-11T21:52:02.293 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:02.285+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:02.313 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:02.305+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:02.433 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:02.427+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:02.437 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:02.432+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:02.452 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:02.447+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:02.487 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:02.480+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:02.498 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:02.493+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:03.080 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:03.073+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:03.081 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:03.075+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:03.272 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:03.265+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:03.297 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:03.291+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:03.429 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:03.420+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:03.430 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:03.424+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:03.469 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:03.462+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:03.528 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:03.521+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:03.539 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:03.532+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:04.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:04.114+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:04.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:04.116+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:04.263 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:04.255+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:04.353 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:04.346+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:04.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:04.448+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:04.465 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:04.459+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:04.471 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:04.463+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:04.494 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:04.487+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:04.601 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:04.573+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:05.078 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:05.071+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:05.079 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:05.073+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:05.315 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:05.307+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:05.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:05.357+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:05.437 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:05.430+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:05.453 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:05.446+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:05.460 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:05.452+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:05.461 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:05.455+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:05.575 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:05.567+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:06.091 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:06.083+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:06.092 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:06.085+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:06.359 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:06.351+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:06.404 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:06.393+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:06.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:06.402+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:06.409 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:06.404+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:06.455 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:06.447+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:06.456 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:06.450+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:06.605 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:06.598+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:07.081 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:52:07.074+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e189 v189) 2019-12-11T21:52:07.099 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:07.093+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:07.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:07.095+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:07.333 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:07.326+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:07.392 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:07.386+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:07.436 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:07.422+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:07.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:07.437+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:07.506 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:07.499+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:07.510 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:07.504+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:07.607 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:07.602+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:08.137 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:08.131+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:08.140 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:08.134+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:08.333 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:08.326+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:08.372 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:08.366+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:08.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:08.438+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:08.470 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:08.464+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:08.500 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:08.492+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:08.501 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:08.495+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:08.640 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:08.633+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:09.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:09.097+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:09.106 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:09.100+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:09.367 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:09.359+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:09.375 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:09.368+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:09.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:09.438+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:09.511 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:09.505+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:09.517 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:09.508+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:09.518 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:09.512+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:09.641 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:09.634+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:10.144 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:10.139+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:10.147 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:10.142+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:10.371 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:10.365+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:10.372 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:10.364+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:10.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:10.429+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:10.465 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:10.459+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:10.481 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:10.474+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:10.484 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:10.479+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:10.604 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:10.598+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:11.148 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:11.143+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:11.153 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:11.148+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:11.369 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:52:11.365+0000 1694f700 -1 osd.0 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:34.730011+0000 front 2019-12-11T21:50:34.725217+0000 (oldest deadline 2019-12-11T21:51:18.224054+0000) 2019-12-11T21:52:11.425 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:11.417+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:11.438 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:52:11.426+0000 1694f700 -1 osd.7 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.119293+0000 front 2019-12-11T21:50:37.274611+0000 (oldest deadline 2019-12-11T21:51:19.994999+0000) 2019-12-11T21:52:11.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:52:11.452+0000 1694f700 -1 osd.2 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:37.723497+0000 front 2019-12-11T21:50:37.754958+0000 (oldest deadline 2019-12-11T21:51:19.423316+0000) 2019-12-11T21:52:11.493 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:11.485+0000 1694f700 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:50:55.905872+0000 (oldest deadline 2019-12-11T21:51:35.905872+0000) 2019-12-11T21:52:11.495 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:52:11.488+0000 1694f700 -1 osd.5 189 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:104482 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e188) 2019-12-11T21:52:11.606 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:52:11.599+0000 1694f700 -1 osd.6 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:32.964651+0000 front 2019-12-11T21:50:32.988466+0000 (oldest deadline 2019-12-11T21:51:18.698640+0000) 2019-12-11T21:52:12.127 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:12.121+0000 1694f700 -1 osd.3 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:36.059537+0000 front 2019-12-11T21:50:36.624337+0000 (oldest deadline 2019-12-11T21:51:24.845286+0000) 2019-12-11T21:52:12.130 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:12.126+0000 1694f700 -1 osd.3 189 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:105176 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384078192 gen 1] snapc 0=[] ondisk+write+known_if_redirected e185) 2019-12-11T21:52:12.151 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:52:12.143+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e189 v189) 2019-12-11T21:52:12.416 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:52:12.409+0000 1694f700 -1 osd.4 189 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:50:35.235953+0000 front 2019-12-11T21:50:35.137311+0000 (oldest deadline 2019-12-11T21:51:20.935906+0000) 2019-12-11T21:52:13.138 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:52:13.130+0000 1694f700 -1 osd.3 190 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105239 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 384078192 gen 2] snapc 0=[] ondisk+write+known_if_redirected e189) 2019-12-11T21:52:14.437 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T21:52:14.404+0000 20e25700 -1 RGWWatcher::handle_error cookie 384078192 err (107) Transport endpoint is not connected 2019-12-11T21:53:08.699 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:53:08.628+0000 36790700 -1 osd.5 195 build_incremental_map_msg missing incremental map 177 2019-12-11T21:53:10.243 INFO:teuthology.orchestra.run.smithi079.stderr:testCopy (test.functional.tests.TestFile) ... ok 2019-12-11T21:53:33.096 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:33.089+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:34.125 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:34.119+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:34.386 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:34.379+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:34.569 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:34.563+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:34.603 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:34.597+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:35.093 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:35.086+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:35.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:35.245+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:35.396 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:35.390+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:35.575 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:35.569+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:35.609 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:35.602+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:35.687 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:35.680+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:36.128 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:36.122+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:36.243 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:36.237+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:36.358 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:36.351+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:36.606 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:36.597+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:36.634 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:36.617+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:36.693 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:36.688+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:37.125 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:37.119+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:37.202 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:37.196+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:37.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:37.282+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:37.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:37.407+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:37.634 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:37.627+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:37.655 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:37.650+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:37.720 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:37.713+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:38.085 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:38.080+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:38.178 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:38.173+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:38.291 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:38.286+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:38.426 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:38.421+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:38.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:38.603+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:38.661 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:38.656+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:38.772 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:38.767+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:39.093 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:39.085+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:39.165 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:39.158+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:39.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:39.252+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:39.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:39.460+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:39.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:39.578+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:39.637 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:39.632+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:39.751 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:39.747+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:40.122 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:40.117+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:40.130 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:40.124+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:40.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:40.263+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:40.456 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:40.452+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:40.603 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:40.598+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:40.620 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:40.615+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:40.803 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:40.797+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:41.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:41.120+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:41.177 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:41.171+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:41.306 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:41.300+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:41.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:41.453+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:41.630 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:41.625+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:41.672 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:41.667+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:41.847 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:41.840+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:42.116 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:42.110+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:42.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:42.205+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:42.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:42.324+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:42.468 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:42.463+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:42.615 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:42.610+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:42.725 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:42.720+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:42.892 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:42.887+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:43.138 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:43.132+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:43.170 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:43.164+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:43.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:43.329+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:43.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:43.466+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:43.601 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:43.596+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:43.718 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:43.712+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:43.922 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:43.918+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:44.159 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:44.153+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:44.179 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:44.173+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:44.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:44.342+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:44.452 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:44.446+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:44.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:44.602+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:44.761 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:44.755+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:44.898 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:44.893+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:45.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:45.131+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:45.193 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:45.186+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:45.378 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:45.372+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:45.413 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:45.408+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:45.644 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:45.639+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:45.750 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:45.745+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:45.914 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:45.908+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:46.092 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:46.086+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:46.169 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:46.164+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:46.413 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:46.409+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:46.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:46.422+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:46.623 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:46.618+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:46.761 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:46.752+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:46.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:46.897+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:47.139 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:47.134+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:47.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:47.207+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:47.392 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:47.386+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:47.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:47.415+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:47.648 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:47.643+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:47.740 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:47.733+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:47.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:47.869+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:48.152 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:48.147+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:48.262 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:48.256+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:48.353 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:48.348+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:48.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:48.468+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:48.612 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:48.607+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:48.693 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:48.688+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:48.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:48.898+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:49.119 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:49.113+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:49.229 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:49.225+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:49.394 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:49.389+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:49.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:49.425+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:49.636 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:49.632+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:49.667 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:49.661+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:49.862 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:49.857+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:50.126 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:50.120+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:50.228 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:50.223+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:50.404 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:50.397+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:50.417 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:50.412+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:50.629 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:50.623+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:50.635 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:50.631+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:50.871 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:50.866+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:51.095 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:51.090+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:51.221 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:51.215+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:51.369 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:51.364+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:51.514 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:51.403+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:51.607 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:51.602+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:51.655 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:51.650+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:51.924 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:51.918+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:52.098 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:52.092+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:52.273 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:52.267+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:52.362 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:52.357+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:52.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:52.403+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:52.604 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:52.600+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:52.695 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:52.690+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:52.930 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:52.924+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:53.088 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:53.083+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:53.263 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:53.258+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:53.381 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:53.375+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:53.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:53.386+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:53.404 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:53.395+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:53:53.635 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:53.629+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:53.700 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:53.694+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:53.911 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:53.906+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:54.058 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:54.052+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:54.317 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:54.311+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:54.369 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:54.363+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:54.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:54.440+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:54.447 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:54.441+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:53:54.661 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:54.656+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:54.748 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:54.743+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:54.866 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:54.861+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:55.043 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:55.037+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:55.355 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:55.348+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:55.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:55.388+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:55.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:55.466+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:55.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:55.468+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:53:55.682 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:55.677+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:55.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:55.794+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:55.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:55.836+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:56.023 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:56.017+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:56.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:56.342+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:56.436 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:56.431+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:56.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:56.515+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:56.522 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:56.517+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:53:56.657 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:56.647+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:56.778 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:56.763+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:56.838 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:56.822+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:57.045 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:57.035+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:57.310 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:57.302+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:57.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:57.465+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:57.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:57.561+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:57.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:57.562+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:53:57.686 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:57.681+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:57.799 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:57.794+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:57.810 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:57.805+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:58.057 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:58.051+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:58.363 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:58.356+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:58.492 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:58.487+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:58.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:58.560+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:58.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:58.561+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:53:58.686 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:58.681+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:58.780 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:58.775+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:53:58.823 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:58.818+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:59.066 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:53:59.060+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:53:59.366 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:53:59.359+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:53:59.507 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:53:59.501+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:53:59.550 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:59.544+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:53:59.551 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:53:59.545+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:53:59.659 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:53:59.654+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:53:59.806 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:53:59.800+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:53:59.814 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:53:59.809+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:00.057 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:00.052+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:00.414 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:00.408+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:00.506 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:00.501+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:00.567 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:00.560+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:00.567 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:00.561+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:00.655 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:54:00.650+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:54:00.832 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:00.827+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:00.853 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:54:00.847+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:54:01.042 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:01.035+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:01.436 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:01.430+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:01.547 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:01.540+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:01.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:01.579+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:01.586 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:01.581+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:01.663 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:54:01.659+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:54:01.797 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:01.791+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:01.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:54:01.898+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:54:02.094 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:02.088+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:02.446 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:02.440+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:02.532 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:02.527+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:02.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:02.578+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:02.586 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:02.581+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:02.698 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:54:02.693+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:54:02.767 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:02.762+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:02.939 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:54:02.933+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:54:03.096 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:03.089+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:03.447 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:03.440+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:03.496 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:03.490+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:03.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:03.595+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:03.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:03.597+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:03.749 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:54:03.744+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:54:03.814 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:03.809+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:03.905 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:54:03.887+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:54:04.082 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:04.076+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:04.471 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:04.465+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:04.487 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:04.482+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:04.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:04.584+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:04.592 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:04.587+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:04.735 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:54:04.730+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:54:04.816 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:04.810+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:04.898 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:54:04.893+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:54:05.057 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:05.050+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:05.443 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:05.436+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:05.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:05.465+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:05.613 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:05.606+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:05.614 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:05.608+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:05.721 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:54:05.715+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:54:05.858 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:54:05.852+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:54:05.866 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:05.860+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:06.106 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:06.101+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:06.463 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:06.456+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:06.500 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:06.495+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:06.616 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:06.610+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:06.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:06.612+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:06.700 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:54:06.695+0000 1694f700 -1 osd.1 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.625216+0000 front 2019-12-11T21:52:51.626509+0000 (oldest deadline 2019-12-11T21:53:33.802933+0000) 2019-12-11T21:54:06.843 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:54:06.838+0000 1694f700 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.266812+0000 front 2019-12-11T21:52:51.630725+0000 (oldest deadline 2019-12-11T21:53:35.065180+0000) 2019-12-11T21:54:06.891 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:06.886+0000 1694f700 -1 osd.2 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:50.187361+0000 front 2019-12-11T21:52:51.625045+0000 (oldest deadline 2019-12-11T21:53:33.537321+0000) 2019-12-11T21:54:07.079 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:54:07.072+0000 1694f700 -1 osd.6 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:51.627984+0000 front 2019-12-11T21:52:52.870886+0000 (oldest deadline 2019-12-11T21:53:32.812805+0000) 2019-12-11T21:54:07.449 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:54:07.443+0000 1694f700 -1 osd.4 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.870419+0000 front 2019-12-11T21:52:52.884118+0000 (oldest deadline 2019-12-11T21:53:36.613948+0000) 2019-12-11T21:54:07.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:07.504+0000 1694f700 -1 osd.0 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.064593+0000 front 2019-12-11T21:52:51.267440+0000 (oldest deadline 2019-12-11T21:53:33.429493+0000) 2019-12-11T21:54:07.534 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:54:07.525+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 41sec e195 v195) 2019-12-11T21:54:07.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:07.625+0000 1694f700 -1 osd.7 195 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:52:52.400760+0000 front 2019-12-11T21:52:52.245384+0000 (oldest deadline 2019-12-11T21:53:34.493252+0000) 2019-12-11T21:54:07.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:07.628+0000 1694f700 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:08.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:08.608+0000 1694f700 -1 osd.7 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:09.686 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:54:09.681+0000 1694f700 -1 osd.7 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105471 2.30s0 2:0d045159:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_8822ea6ed56246b5881c2b8c298641a7:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e195) 2019-12-11T21:54:15.590 INFO:teuthology.orchestra.run.smithi079.stderr:testCopy404s (test.functional.tests.TestFile) ... ok 2019-12-11T21:54:32.012 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:22:18.941 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.028 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:22:18.959 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.033 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:22:18.963 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.036 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:22:18.967 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.040 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:22:18.942 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.053 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:22:18.955 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.224 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:22:19.155 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.295 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:22:19.225 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:32.303 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:22:19.204 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:54:43.443 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:43.369+0000 37792700 -1 osd.0 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.490 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:43.477+0000 34f8d700 -1 osd.2 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.601 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:43.549+0000 31f87700 -1 osd.2 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:43.551+0000 31f87700 -1 osd.2 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.615 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:43.400+0000 37792700 -1 osd.0 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.623 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:43.435+0000 31786700 -1 osd.0 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.626 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:43.572+0000 35f8f700 -1 osd.2 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.633 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:43.461+0000 31f87700 -1 osd.0 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.882 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:54:43.797+0000 38794700 -1 osd.2 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:54:43.957 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:54:43.873+0000 38794700 -1 osd.0 199 build_incremental_map_msg missing incremental map 199 2019-12-11T21:55:00.291 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:00.286+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:01.305 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:01.300+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:02.108 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:02.102+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:02.284 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:02.279+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:03.127 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:03.120+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:03.269 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:03.264+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:04.145 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:04.140+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:04.321 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:04.316+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:05.155 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:05.150+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:05.286 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:05.281+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:06.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:06.159+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:06.167 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:06.163+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:06.255 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:06.248+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:06.269 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:06.265+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:07.192 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:07.187+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:07.193 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:07.189+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:07.238 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:07.234+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:07.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:07.257+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:07.714 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:07.706+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:08.169 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:08.162+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:08.172 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:08.167+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:08.194 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:08.190+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:08.247 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:08.240+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:08.312 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:08.305+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:08.691 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:08.683+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:09.079 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:09.071+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:09.152 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:09.147+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:09.154 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:09.150+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:09.236 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:09.232+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:09.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:09.264+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:09.345 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:09.339+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:09.659 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:09.651+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:10.122 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:10.113+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:10.197 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:10.191+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:10.197 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:10.193+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:10.215 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:10.210+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:10.216 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:10.211+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:10.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:10.282+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:10.305 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:10.299+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:10.714 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:10.707+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:11.090 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:11.082+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:11.217 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:11.211+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:11.218 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:11.213+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:11.250 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:11.244+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:11.250 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:11.246+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:11.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:11.251+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:11.272 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:11.266+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:11.693 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:11.686+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:12.100 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:12.093+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:12.232 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:12.224+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:12.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:12.229+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:12.246 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:12.240+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:12.248 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:12.244+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:12.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:12.264+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:12.293 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:12.286+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:12.739 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:12.732+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:13.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:13.057+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:13.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:13.229+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:13.236 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:13.232+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:13.280 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:13.275+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:13.280 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:13.276+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:13.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:13.280+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:13.341 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:13.334+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:13.769 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:13.762+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:14.082 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:14.076+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:14.255 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:14.250+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:14.256 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:14.251+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:14.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:14.284+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:14.331 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:14.325+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:14.331 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:14.327+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:14.335 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:14.330+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:14.744 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:14.734+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:15.106 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:15.099+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:15.235 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:15.228+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:15.237 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:15.231+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:15.288 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:15.282+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:15.289 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:15.284+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:15.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:15.305+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:15.371 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:15.366+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:15.768 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:15.761+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:15.905 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:15.900+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:16.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:16.119+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:16.247 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:16.241+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:16.248 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:16.243+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:16.249 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:16.243+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:16.250 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:16.245+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:16.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:16.263+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:16.412 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:16.407+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:16.730 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:16.723+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:16.883 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:16.878+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:17.138 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:17.131+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:17.244 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:17.238+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:17.244 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:17.240+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:17.260 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:17.255+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:17.261 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:17.256+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:17.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:17.276+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:17.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:17.409+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:17.728 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:17.722+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:17.879 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:17.874+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:18.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:18.120+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:18.215 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:18.211+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:18.216 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:18.212+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:18.293 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:18.288+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:18.294 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:18.289+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:18.328 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:18.322+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:18.377 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:18.372+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:18.741 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:18.735+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:18.931 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:18.925+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:19.170 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:19.163+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:19.231 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:19.225+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:19.232 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:19.227+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:19.305 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:19.297+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:19.316 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:19.310+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:19.316 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:19.312+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:19.413 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:19.407+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:19.699 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:19.692+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:19.904 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:19.898+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:20.182 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:20.175+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:20.272 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:20.266+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:20.272 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:20.268+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:20.331 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:20.323+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:20.340 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:20.333+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:20.340 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:20.336+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:20.427 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:20.421+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:20.736 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:20.729+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:20.926 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:20.919+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:21.146 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:21.138+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:21.275 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:21.270+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:21.276 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:21.271+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:21.373 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:21.366+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:21.374 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:21.369+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:21.374 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:21.368+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:21.386 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:21.381+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:21.763 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:21.756+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:21.947 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:21.940+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:21.948 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:21.942+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:22.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:22.145+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:22.310 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:22.304+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:22.313 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:22.309+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:22.402 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:22.396+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:22.403 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:22.399+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:22.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:22.403+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:22.427 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:22.421+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:22.723 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:22.716+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:22.961 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:22.953+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:22.966 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:22.960+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:23.123 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:23.115+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:23.311 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:23.306+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:23.311 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:23.307+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:23.387 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:23.381+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:23.388 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:23.383+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:23.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:23.412+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:23.451 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:23.445+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:23.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:23.714+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:24.008 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:23.997+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:24.009 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:24.001+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:24.119 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:24.111+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:24.340 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:24.332+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:24.340 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:24.334+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:24.434 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:24.429+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:24.434 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:24.426+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:24.436 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:24.432+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:24.450 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:24.444+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:24.694 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:24.687+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:25.049 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:25.036+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:25.049 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:25.039+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:25.117 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:25.110+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:25.382 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:25.376+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:25.383 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:25.378+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:25.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:25.382+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:25.430 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:25.424+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:25.431 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:25.427+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:25.497 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:25.492+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:25.692 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:25.685+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:26.061 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:26.055+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:26.065 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:26.059+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:26.091 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:26.084+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:26.349 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:26.341+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:26.349 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:26.342+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:26.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:26.359+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:26.447 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:26.435+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:26.448 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:26.436+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:26.528 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:26.520+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:26.736 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:26.728+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:27.092 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:27.086+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:27.096 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:27.091+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:27.118 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:27.111+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:27.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:27.381+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:27.390 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:27.384+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:27.391 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:27.386+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:27.437 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:27.431+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:27.441 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:27.436+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:27.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:27.504+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:27.770 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:27.763+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:28.113 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:28.106+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:28.116 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:28.111+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:28.120 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:28.114+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:28.349 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:28.342+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:28.384 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:28.377+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:28.385 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:28.380+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:28.493 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:28.486+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:28.493 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:28.489+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:28.554 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:28.547+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:28.731 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:28.724+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:29.103 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:29.096+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:29.106 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:29.101+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:29.156 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:29.147+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:29.382 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:29.376+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:29.383 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:29.378+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:29.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:29.386+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:29.521 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:29.516+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:29.522 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:29.517+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:29.575 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:29.569+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:29.740 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:29.733+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:30.113 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:30.106+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:30.116 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:30.111+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:30.208 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:30.201+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:30.415 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:30.409+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:30.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:30.414+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:30.422 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:30.417+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:30.567 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:30.562+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:30.569 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:30.565+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:30.614 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:30.608+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:30.769 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:30.762+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:31.164 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:31.153+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:31.165 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:31.156+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:31.241 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:31.234+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:31.392 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:31.384+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:31.404 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:31.397+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:31.405 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:31.400+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:31.583 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:31.577+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:31.603 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:31.597+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:31.618 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:31.612+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:31.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:31.737+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:32.200 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:32.192+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:32.201 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:32.195+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:32.242 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:32.234+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:32.347 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:32.341+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:32.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:32.392+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:32.399 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:32.396+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:32.561 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:32.556+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:32.650 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:32.643+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:32.651 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:32.645+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:32.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:32.713+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:33.186 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:33.180+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:33.190 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:33.185+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:33.198 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:33.190+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:33.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:33.378+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:33.404 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:33.399+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:33.405 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:33.401+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:33.529 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:33.524+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:33.612 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:33.607+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:33.613 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:33.608+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:33.692 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:33.685+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:34.178 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:34.173+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:34.184 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:34.175+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:34.214 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:34.208+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:34.399 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:34.393+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:34.400 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:34.395+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:34.401 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:34.394+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:34.579 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:34.572+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:34.593 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:34.586+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:34.595 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:34.589+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:34.698 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:34.690+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:35.218 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:35.212+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:35.222 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:35.217+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:35.235 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:35.228+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:35.424 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:35.416+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:35.425 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:35.420+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:35.426 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:35.422+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:35.579 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:35.573+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:35.611 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:35.606+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:35.612 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:35.608+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:35.664 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:35.657+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:36.218 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:36.211+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:36.263 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:36.256+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:36.266 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:36.261+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:36.429 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:36.423+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:36.430 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:36.425+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:36.479 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:36.469+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:36.598 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:36.592+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:36.599 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:36.594+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:36.619 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:36.614+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:36.661 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:36.654+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:37.214 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:37.206+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:37.297 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:37.289+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:37.297 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:37.291+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:37.433 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:37.428+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:37.434 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:37.430+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:37.491 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:37.484+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:37.634 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:37.628+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:37.637 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:37.633+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:37.656 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:37.650+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:37.679 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:37.672+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:38.240 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:38.233+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:38.251 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:38.243+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:38.252 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:38.246+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:38.412 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:38.405+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:38.413 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:38.408+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:38.502 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:38.495+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:38.653 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:38.647+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:38.656 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:38.651+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:38.691 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:38.685+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:38.724 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:38.717+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:39.224 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:39.217+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:39.227 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:39.222+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:39.272 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:39.265+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:39.464 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:39.459+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:39.465 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:39.460+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:39.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:39.514+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:39.617 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:39.612+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:39.619 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:39.615+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:39.702 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:39.696+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:39.719 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:39.712+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:40.184 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:40.178+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:40.188 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:40.183+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:40.300 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:40.294+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:40.426 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:40.420+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:40.427 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:40.424+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:40.492 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:40.486+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:40.672 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:40.666+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:40.673 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:40.668+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:40.750 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:40.745+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:40.758 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:40.752+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:41.204 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:41.198+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:41.205 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:41.200+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:41.336 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:41.329+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:41.444 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:41.439+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:41.447 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:41.443+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:41.481 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:41.474+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:41.641 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:41.635+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:41.642 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:41.636+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:41.754 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:41.748+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:41.785 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:41.779+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:42.215 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:42.207+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:42.219 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:42.215+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:42.364 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:42.358+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:42.414 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:42.408+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:42.416 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:42.411+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:42.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:42.514+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:42.624 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:42.618+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:42.628 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:42.622+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:42.733 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:42.727+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:42.771 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:42.764+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:42.930 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:55:42.923+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e201 v201) 2019-12-11T21:55:43.246 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:43.241+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:43.249 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:43.245+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:43.339 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:43.333+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:43.455 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:43.450+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:43.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:43.452+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:43.483 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:43.477+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:43.607 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:43.600+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:43.608 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:43.602+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:43.700 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:43.693+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:43.767 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:43.759+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:44.260 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:44.253+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:44.261 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:44.256+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:44.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:44.343+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:44.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:44.471+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:44.480 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:44.474+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:44.528 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:44.522+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:44.610 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:44.601+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:44.611 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:44.604+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:44.682 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:44.676+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:44.789 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:44.783+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:45.217 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:45.210+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:45.217 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:45.212+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:45.308 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:45.302+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:45.504 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:45.498+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:45.514 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:45.507+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:45.514 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:45.509+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:45.594 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:45.588+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:45.595 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:45.589+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:45.670 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:45.663+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:45.766 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:45.759+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:46.219 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:46.214+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:46.222 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:46.218+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:46.316 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:46.311+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:46.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:46.471+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:46.480 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:46.474+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:46.541 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:46.535+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:46.547 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:46.541+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:46.551 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:46.545+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:46.652 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:46.646+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:46.750 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:46.743+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:47.264 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:47.258+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:47.268 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:47.263+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:47.337 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:47.330+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:47.460 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:47.452+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:47.460 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:47.454+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:47.552 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:47.545+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:47.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:47.547+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:47.587 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:47.580+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:47.640 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:47.629+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:47.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:47.727+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:47.960 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:55:47.954+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e201 v201) 2019-12-11T21:55:48.248 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:48.240+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:48.251 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:48.243+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:48.307 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:48.299+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:48.505 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:48.501+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:48.506 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:48.502+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:48.566 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:48.560+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:48.566 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:48.562+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:48.602 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:48.595+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:48.673 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:48.666+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:48.692 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:48.688+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:49.257 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:49.250+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:49.260 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:49.255+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:49.324 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:49.316+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:49.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:49.459+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:49.464 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:49.460+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:49.535 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:49.530+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:49.537 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:49.533+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:49.640 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:49.632+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:49.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:49.717+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:49.747 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:49.741+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:50.287 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:50.261+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:50.289 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:50.264+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:50.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:50.296+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:50.496 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:50.492+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:50.498 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:50.494+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:50.528 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:50.520+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:50.529 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:50.524+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:50.636 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:50.629+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:50.710 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:50.702+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:50.767 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:50.762+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:51.231 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:51.223+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:51.232 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:51.226+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:51.320 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:51.308+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:51.490 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:51.484+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:51.491 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:51.487+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:51.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:51.527+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:51.535 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:51.530+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:51.634 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:51.627+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:51.733 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:51.725+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:51.762 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:51.757+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:52.266 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:52.256+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:52.266 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:52.259+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:52.293 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:52.285+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:52.461 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:52.456+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:52.466 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:52.461+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:52.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:52.578+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:52.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:52.580+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:52.644 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:52.636+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:52.719 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:52.712+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:52.807 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:52.802+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:52.982 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:55:52.973+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e201 v201) 2019-12-11T21:55:53.252 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:53.245+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:53.317 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:53.309+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:53.317 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:53.312+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:53.435 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:53.428+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:53.436 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:53.431+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:53.582 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:53.575+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:53.583 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:53.578+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:53.667 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:53.659+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:53.690 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:53.683+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:53.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:53.785+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:54.295 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:54.288+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:54.296 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:54.291+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:54.299 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:54.293+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:54.398 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:54.391+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:54.400 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:54.394+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:54.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:54.575+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:54.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:54.579+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:54.638 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:54.631+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:54.726 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:54.718+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:54.825 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:54.819+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:55.266 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:55.259+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:55.267 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:55.261+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:55.305 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:55.298+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:55.365 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:55.357+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:55.366 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:55.360+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:55.560 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:55.553+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:55.561 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:55.556+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:55.647 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:55.638+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:55.777 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:55.770+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:55.796 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:55.790+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:56.260 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:56.252+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:56.261 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:56.254+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:56.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:56.328+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:56.390 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:56.383+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:56.391 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:56.386+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:56.540 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:56.535+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:56.540 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:56.536+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:56.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:56.617+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:56.785 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:56.771+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:56.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:56.807+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:57.260 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:57.251+0000 1694f700 -1 osd.5 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:54:41.455809+0000 (oldest deadline 2019-12-11T21:55:21.455809+0000) 2019-12-11T21:55:57.264 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:57.254+0000 1694f700 -1 osd.5 201 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:57.352 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:55:57.345+0000 1694f700 -1 osd.4 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.874901+0000 front 2019-12-11T21:54:24.143477+0000 (oldest deadline 2019-12-11T21:55:08.823687+0000) 2019-12-11T21:55:57.442 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:57.434+0000 1694f700 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:24.993091+0000 front 2019-12-11T21:54:24.387305+0000 (oldest deadline 2019-12-11T21:55:09.524420+0000) 2019-12-11T21:55:57.444 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:57.437+0000 1694f700 -1 osd.3 201 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:57.577 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:57.571+0000 1694f700 -1 osd.2 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.833124+0000 front 2019-12-11T21:54:21.922195+0000 (oldest deadline 2019-12-11T21:55:05.311401+0000) 2019-12-11T21:55:57.578 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:57.574+0000 1694f700 -1 osd.2 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:57.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:55:57.598+0000 1694f700 -1 osd.7 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.647398+0000 front 2019-12-11T21:54:22.548383+0000 (oldest deadline 2019-12-11T21:55:06.020455+0000) 2019-12-11T21:55:57.817 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:55:57.810+0000 1694f700 -1 osd.6 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:21.839393+0000 front 2019-12-11T21:54:21.767081+0000 (oldest deadline 2019-12-11T21:55:07.475952+0000) 2019-12-11T21:55:57.838 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:55:57.833+0000 1694f700 -1 osd.0 201 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:54:22.905068+0000 front 2019-12-11T21:54:22.903118+0000 (oldest deadline 2019-12-11T21:55:08.185099+0000) 2019-12-11T21:55:58.022 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:55:58.016+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e201 v201) 2019-12-11T21:55:58.606 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:58.535+0000 1694f700 -1 osd.3 202 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:58.781 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:58.727+0000 1694f700 -1 osd.2 202 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:58.823 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:55:58.714+0000 1694f700 -1 osd.5 202 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e199) 2019-12-11T21:55:59.680 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:55:59.675+0000 1694f700 -1 osd.3 203 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:105596 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384091728 gen 1] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:55:59.801 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:55:59.786+0000 1694f700 -1 osd.2 203 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:105601 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~8265 in=8265b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e197) 2019-12-11T21:56:00.330 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T21:56:00.285+0000 20e25700 -1 RGWWatcher::handle_error cookie 384091728 err (107) Transport endpoint is not connected 2019-12-11T21:56:01.166 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyDestinationSlashProblems (test.functional.tests.TestFile) ... ok 2019-12-11T21:56:27.607 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:56:27.597+0000 3578e700 -1 osd.2 206 build_incremental_map_msg missing incremental map 176 2019-12-11T21:56:27.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:56:27.598+0000 3578e700 -1 osd.2 206 build_incremental_map_msg missing incremental map 199 2019-12-11T21:56:55.774 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:56:55.767+0000 38f95700 -1 osd.0 206 build_incremental_map_msg missing incremental map 176 2019-12-11T21:56:55.775 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:56:55.768+0000 38f95700 -1 osd.0 206 build_incremental_map_msg missing incremental map 199 2019-12-11T21:57:01.982 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:01.978+0000 3578e700 -1 osd.1 206 build_incremental_map_msg missing incremental map 172 2019-12-11T21:57:01.989 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:01.985+0000 3578e700 -1 osd.1 206 build_incremental_map_msg missing incremental map 183 2019-12-11T21:57:18.330 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyFromHeader (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:21.648 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:21.643+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:22.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:22.105+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:22.137 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:22.132+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:22.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:22.138+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:22.681 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:22.675+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:23.144 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:23.140+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:23.161 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:23.156+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:23.166 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:23.160+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:23.688 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:23.682+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:24.153 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:24.146+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:24.166 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:24.161+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:24.210 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:24.204+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:24.685 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:24.679+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:25.024 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:25.016+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:25.122 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:25.118+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:25.221 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:25.213+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:25.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:25.218+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:25.225 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:25.219+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:25.246 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:25.241+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:25.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:25.726+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:26.032 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:26.026+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:26.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:26.161+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:26.204 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:26.196+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:26.232 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:26.227+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:26.253 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:26.248+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:26.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:26.261+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyFromHeader404s (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:26.783 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:26.778+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:27.006 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:26.999+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:27.123 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:27.117+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:27.163 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:27.156+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:27.222 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:27.217+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:27.230 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:27.225+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:27.256 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:27.249+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:27.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:27.808+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:28.025 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:28.019+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:28.130 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:28.126+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:28.159 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:28.154+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:28.185 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:28.179+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:28.246 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:28.241+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:28.283 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyNoDestinationHeader (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:28.285 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:28.278+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:28.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:28.806+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:29.079 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:29.073+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:29.103 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:29.097+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:29.138 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:29.133+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:29.154 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:29.148+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:29.211 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:29.204+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:29.249 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:29.244+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:29.788 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:29.783+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:30.071 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:30.065+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:30.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:30.107+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:30.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:30.148+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:30.188 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:30.183+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:30.208 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:30.204+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:30.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:30.238+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:30.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:30.805+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:31.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:31.087+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:31.132 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:31.126+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:31.164 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:31.155+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:31.176 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:31.170+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:31.188 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:31.182+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:31.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:31.244+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:31.781 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:31.775+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:32.133 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:32.127+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:32.142 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:32.137+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:32.165 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:32.159+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:32.198 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:32.191+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:32.199 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:32.194+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:32.275 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:32.268+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:32.769 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:32.764+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:33.131 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:33.126+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:33.145 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:33.138+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:33.156 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:33.150+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:33.168 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:33.163+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:33.206 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:33.198+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:33.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:33.315+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:33.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:33.730+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:34.147 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:34.141+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:34.154 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:34.149+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:34.172 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:34.167+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:34.186 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:34.180+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:34.233 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:34.227+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:34.314 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:34.308+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:34.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:34.705+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:35.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:35.111+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:35.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:35.139+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:35.150 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:35.144+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:35.193 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:35.188+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:35.194 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:35.190+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:35.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:35.346+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:35.674 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:35.668+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:36.124 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:36.117+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:36.134 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:36.127+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:36.187 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:36.182+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:36.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:36.206+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:36.233 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:36.228+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:36.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:36.381+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:36.639 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:36.632+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:37.093 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:37.088+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:37.107 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:37.101+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:37.204 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:37.199+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:37.221 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:37.216+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:37.228 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:37.223+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:37.421 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:37.415+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:37.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:37.594+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:38.073 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:38.067+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:38.103 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:38.098+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:38.204 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:38.198+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:38.205 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:38.197+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:38.246 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:38.240+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:38.457 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:38.448+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:38.647 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:38.641+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:39.073 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:39.067+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:39.097 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:39.088+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:39.182 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:39.176+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:39.245 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:39.240+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:39.277 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:39.272+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:39.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:39.495+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:39.688 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:39.682+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:40.045 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:40.041+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:40.131 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:40.126+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:40.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:40.144+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:40.252 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:40.246+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:40.261 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:40.255+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:40.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:40.547+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:40.663 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:40.657+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:41.024 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:41.018+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:41.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:41.153+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:41.166 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:41.160+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:41.289 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:41.283+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:41.309 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:41.301+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:41.557 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:41.551+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:41.669 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:41.665+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:42.037 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:42.033+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:42.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:42.147+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:42.204 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:42.197+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:42.266 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:42.261+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:42.330 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:42.315+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:42.556 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:42.549+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:42.713 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:42.707+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:43.039 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:43.035+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:43.114 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:43.107+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:43.253 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:43.247+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:43.254 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:43.248+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:43.333 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:43.325+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:43.532 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:43.525+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:43.733 INFO:teuthology.orchestra.run.smithi079.stderr:testDelete (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:43.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:43.750+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:44.045 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:44.038+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteOfFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:44.090 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:44.084+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:44.219 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:44.213+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:44.297 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:44.292+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:44.304 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:44.299+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:44.571 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:44.566+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:44.727 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:44.722+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:45.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:45.013+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:45.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:45.060+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:45.178 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:45.171+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:45.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:45.320+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:45.347 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:45.342+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:45.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:45.599+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:45.721 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:45.716+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:46.057 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:46.053+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:46.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:46.099+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:46.159 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:46.153+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:46.317 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:46.312+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:46.352 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:46.347+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:46.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:46.582+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:46.767 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:46.761+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:47.049 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:47.040+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:47.074 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:47.068+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:47.209 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:47.202+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:47.353 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:47.347+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:47.359 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:47.353+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:47.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:47.597+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:47.803 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:47.799+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:48.089 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:48.084+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:48.119 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:48.112+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:48.204 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:48.199+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:48.330 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:48.325+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:48.343 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:48.337+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:48.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:48.603+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:48.798 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:48.792+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:49.068 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:49.062+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:49.137 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:49.127+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:49.209 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:49.200+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:49.313 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:49.308+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:49.377 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:49.372+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:49.666 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:49.659+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:49.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:49.753+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:50.078 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:50.073+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:50.178 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:50.171+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:50.240 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:50.233+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:50.272 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:50.267+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:50.372 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:50.367+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:50.699 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:50.687+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:50.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:50.753+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:51.097 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:51.092+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:51.160 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:51.154+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:51.221 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:51.206+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:51.254 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:51.249+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:51.376 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:51.371+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:51.719 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:51.711+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:51.753 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:51.748+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:52.075 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:52.069+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:52.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:52.204+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:52.244 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:52.238+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:52.254 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:52.249+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:52.376 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:52.361+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:52.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:52.727+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:52.741 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:52.734+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:53.056 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:53.050+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:53.208 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:53.201+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:53.243 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:53.237+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:53.304 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:53.299+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:53.384 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:53.377+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:53.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:53.741+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:53.752 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:53.746+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:53.997 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteThen404s (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:54.070 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:54.060+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:54.170 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:54.164+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:54.286 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:54.280+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:54.348 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:54.331+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:54.387 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:54.380+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:54.744 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:54.737+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:54.746 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:54.739+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:55.077 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:55.070+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:55.130 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:55.123+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:55.316 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:55.308+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:55.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:55.322+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:55.383 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:55.377+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:55.703 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:55.697+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:55.729 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:55.723+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:56.044 INFO:teuthology.orchestra.run.smithi079.stderr:testEtagResponse (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:56.083 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:56.076+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:56.125 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:56.116+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:56.299 INFO:teuthology.orchestra.run.smithi079.stderr:testEtagWayoff (test.functional.tests.TestFile) ... ok 2019-12-11T21:57:56.355 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:56.346+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:56.356 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:56.350+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:56.374 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:56.365+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:56.675 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:56.669+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:56.773 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:56.767+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:57.098 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:57.091+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:57.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:57.118+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:57.327 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:57.321+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:57.357 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:57.351+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:57.398 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:57.389+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:57:57.707 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:57:57.701+0000 1694f700 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:37.736561+0000 front 2019-12-11T21:56:41.953851+0000 (oldest deadline 2019-12-11T21:57:21.363310+0000) 2019-12-11T21:57:57.799 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:57:57.793+0000 1694f700 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:35.912246+0000 front 2019-12-11T21:56:35.918731+0000 (oldest deadline 2019-12-11T21:57:21.464351+0000) 2019-12-11T21:57:58.067 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:57:58.057+0000 1694f700 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:43.631090+0000 front 2019-12-11T21:56:37.753866+0000 (oldest deadline 2019-12-11T21:57:21.441305+0000) 2019-12-11T21:57:58.096 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:57:58.090+0000 1694f700 -1 osd.4 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.366544+0000 front 2019-12-11T21:56:39.229386+0000 (oldest deadline 2019-12-11T21:57:24.462699+0000) 2019-12-11T21:57:58.308 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:57:58.301+0000 1694f700 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:40.549545+0000 front 2019-12-11T21:56:40.559025+0000 (oldest deadline 2019-12-11T21:57:24.637555+0000) 2019-12-11T21:57:58.380 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T21:57:58.374+0000 1694f700 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.492885+0000 front 2019-12-11T21:56:39.726059+0000 (oldest deadline 2019-12-11T21:57:21.776789+0000) 2019-12-11T21:57:58.415 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:57:58.406+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e206 v206) 2019-12-11T21:57:58.416 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:57:58.410+0000 1694f700 -1 osd.6 206 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T21:56:39.874187+0000 front 2019-12-11T21:56:39.872649+0000 (oldest deadline 2019-12-11T21:57:24.412858+0000) 2019-12-11T21:58:28.082 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:58:28.078+0000 1694f700 -1 osd.2 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106082 2.21s0 2:8647c74a:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_80179a5e8d3843e1a6572243f6397bcb:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~25669 in=25669b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (58) in=78b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e206) 2019-12-11T21:58:30.145 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:26:17.076 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.153 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:26:17.084 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.158 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:26:17.088 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.162 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:26:17.093 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.167 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:26:17.068 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.179 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:26:17.080 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.403 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:26:17.333 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.409 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:26:17.340 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:30.422 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:26:17.319 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:58:32.367 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:58:32.320+0000 21452700 -1 reset not still connected to 0x3adb6290 2019-12-11T21:58:56.368 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:58:56.358+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:58:57.070 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:58:57.064+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:58:57.352 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:58:57.346+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:58:57.593 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:58:57.586+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:58:58.056 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:58:58.050+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:58:58.387 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:58:58.378+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:58:58.550 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:58:58.542+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:58:59.033 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:58:59.024+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:58:59.397 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:58:59.389+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:58:59.537 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:58:59.528+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:58:59.837 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:58:59.829+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:00.077 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:00.068+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:00.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:00.383+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:00.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:00.400+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:00.533 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:00.525+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:00.890 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:00.882+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:00.973 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:00.968+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:01.098 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:01.090+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:01.405 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:01.373+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:01.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:01.419+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:01.517 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:01.510+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:01.872 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:01.862+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:01.976 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:01.970+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:02.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:02.098+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:02.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:02.403+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:02.436 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:02.430+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:02.570 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:02.563+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:02.859 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:02.851+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:02.957 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:02.952+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:03.108 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:03.101+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:03.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:03.428+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:03.482 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:03.471+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:03.548 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:03.535+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:03.838 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:03.830+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:03.969 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:03.961+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:04.145 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:04.135+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:04.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:04.406+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:04.431 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:04.425+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:04.590 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:04.571+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:04.858 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:04.848+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:05.016 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:05.002+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:05.189 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:05.182+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:05.393 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:05.382+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:05.435 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:05.430+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:05.562 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:05.554+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:05.861 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:05.843+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:05.974 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:05.969+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:06.161 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:06.151+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:06.161 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:06.154+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:06.400 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:06.393+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:06.435 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:06.428+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:06.601 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:06.592+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:06.823 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:06.817+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:06.936 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:06.930+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:07.136 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:07.127+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:07.137 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:07.132+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:07.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:07.380+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:07.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:07.451+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:07.582 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:07.573+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:07.831 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:07.824+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:07.926 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:07.919+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:08.147 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:08.142+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:08.153 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:08.145+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:08.398 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:08.387+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:08.436 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:08.431+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:08.573 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:08.563+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:08.876 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:08.866+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:08.878 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:08.872+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:08.983 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:08.973+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:09.136 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:09.124+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:09.137 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:09.125+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:09.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:09.391+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:09.482 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:09.474+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:09.552 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:09.543+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:09.865 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:09.853+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:09.928 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:09.920+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:09.942 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:09.936+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:10.115 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:10.110+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:10.115 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:10.111+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:10.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:10.422+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:10.508 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:10.503+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:10.575 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:10.568+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:10.873 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:10.865+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:10.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:10.937+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:10.979 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:10.974+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:11.114 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:11.108+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:11.115 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:11.110+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:11.397 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:11.388+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:11.473 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:11.467+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:11.615 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:11.609+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:11.879 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:11.872+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:11.909 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:11.903+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:12.027 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:12.020+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:12.115 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:12.107+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:12.119 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:12.108+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:12.420 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:12.412+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:12.448 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:12.442+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:12.651 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:12.644+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:12.861 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:12.856+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:12.910 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:12.903+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:13.073 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:13.065+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:13.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:13.096+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:13.105 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:13.100+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:13.425 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:13.417+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:13.495 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:13.489+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:13.703 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:13.697+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:13.865 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:13.859+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:13.884 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:13.877+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:14.050 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:14.044+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:14.052 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:14.047+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:14.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:14.095+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:14.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:14.096+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:14.463 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:14.457+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:14.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:14.462+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:14.682 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:14.674+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:14.897 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:14.890+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:14.920 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:14.915+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:15.024 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:15.014+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:15.024 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:15.017+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:15.113 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:15.107+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:15.114 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:15.109+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:15.472 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:15.462+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:15.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:15.477+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:15.708 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:15.701+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:15.886 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:15.879+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:15.899 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:15.890+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:16.025 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:16.017+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:16.027 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:16.019+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:16.131 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:16.123+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:16.131 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:16.125+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:16.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:16.450+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:16.498 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:16.475+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:16.729 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:16.720+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:16.855 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:16.847+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:16.946 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:16.941+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:16.994 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:16.988+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:16.998 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:16.993+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:17.129 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:17.111+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:17.129 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:17.115+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:17.466 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:17.461+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:17.532 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:17.523+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:17.690 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:17.682+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:17.831 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:17.824+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:17.936 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:17.932+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:18.036 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:18.027+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:18.039 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:18.029+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:18.152 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:18.145+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:18.152 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:18.146+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:18.490 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:18.483+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:18.500 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:18.496+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:18.724 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:18.718+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:18.829 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:18.821+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:18.949 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:18.943+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:testFileCreate (test.functional.tests.TestFile) ... ok 2019-12-11T21:59:19.064 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:19.056+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:19.065 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:19.058+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:19.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:19.101+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:19.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:19.103+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:19.460 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:19.455+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:19.499 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:19.489+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:19.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:19.764+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:19.819 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:19.811+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:19.951 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:19.946+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:20.078 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:20.070+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:20.078 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:20.073+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:20.087 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:20.080+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:20.088 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:20.083+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:20.195 INFO:teuthology.orchestra.run.smithi079.stderr:testGetContentType (test.functional.tests.TestFile) ... ok 2019-12-11T21:59:20.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:20.447+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:20.493 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:20.487+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:20.736 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:20.728+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:20.803 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:20.795+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:20.950 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:20.940+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:21.033 INFO:teuthology.orchestra.run.smithi079.stderr:testGetOnFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-11T21:59:21.040 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:21.032+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:21.041 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:21.035+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:21.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:21.116+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:21.122 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:21.118+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:21.466 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:21.459+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:21.468 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:21.463+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:21.767 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:21.756+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:21.847 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:21.839+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:21.965 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:21.958+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:22.089 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:22.081+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:22.090 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:22.084+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:22.170 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:22.164+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:22.172 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:22.167+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:22.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:22.464+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:22.518 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:22.511+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:22.791 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:22.784+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:22.807 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:22.799+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:23.014 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:23.007+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:23.114 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:23.106+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:23.115 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:23.109+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:23.145 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:23.132+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:23.145 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:23.134+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:23.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:23.463+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:23.537 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:23.528+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:23.837 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:23.830+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:23.847 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:23.840+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:24.066 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:24.059+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:24.092 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:24.083+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:24.095 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:24.089+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:24.103 INFO:teuthology.orchestra.run.smithi079.stderr:testHead (test.functional.tests.TestFile) ... ok 2019-12-11T21:59:24.111 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:24.103+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:24.111 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:24.105+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:24.504 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:24.496+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:24.548 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:24.541+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:24.813 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:24.806+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:24.818 INFO:teuthology.orchestra.run.smithi079.stderr:testHeadOnFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-11T21:59:24.871 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:24.863+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:25.036 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:25.030+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:25.067 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:25.061+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:25.068 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:25.062+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:25.074 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:25.068+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:25.074 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:25.069+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:25.491 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:25.484+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:25.559 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:25.552+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:25.802 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:25.796+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:25.873 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:25.865+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:26.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:26.007+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:26.066 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:26.059+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:26.067 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:26.061+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:26.069 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:26.063+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:26.070 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:26.064+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:26.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:26.449+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:26.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:26.545+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:26.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:26.763+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:26.887 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:26.872+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:27.034 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:27.027+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:27.078 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:27.050+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:27.081 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:27.053+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:27.082 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:27.064+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:27.082 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:27.066+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:27.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:27.420+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:27.589 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:27.582+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:27.809 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:27.803+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:27.918 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:27.911+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:27.999 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:27.992+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:28.049 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:28.037+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:28.049 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:28.040+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:28.098 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:28.092+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:28.099 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:28.094+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:28.418 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:28.412+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:28.590 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:28.583+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:28.657 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:59:28.653+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e212 v212) 2019-12-11T21:59:28.809 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:28.804+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:28.947 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:28.942+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:28.996 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:28.990+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:29.013 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:29.007+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:29.013 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:29.009+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:29.053 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:29.047+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:29.054 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:29.049+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:29.384 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:29.377+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:29.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:29.544+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:29.803 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:29.795+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:29.981 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:29.976+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:30.031 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:30.025+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:30.038 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:30.031+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:30.040 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:30.034+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:30.040 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:30.035+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:30.041 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:30.037+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:30.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:30.403+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:30.506 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:30.501+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:30.769 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:30.763+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:30.937 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:30.931+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:30.996 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:30.991+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:31.045 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:31.040+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:31.055 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:31.049+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:31.091 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:31.071+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:31.092 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:31.073+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:31.454 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:31.449+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:31.511 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:31.506+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:31.747 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:31.742+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:31.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:31.962+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:32.045 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:32.038+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:32.046 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:32.041+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:32.052 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:32.043+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:32.056 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:32.049+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:32.057 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:32.053+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:32.485 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:32.478+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:32.531 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:32.527+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:32.789 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:32.783+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:33.006 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:33.000+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:33.008 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:33.004+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:33.023 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:33.018+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:33.085 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:33.073+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:33.085 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:33.076+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:33.101 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:33.095+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:33.465 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:33.458+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:33.503 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:33.497+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:33.783 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:33.776+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:33.826 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:59:33.817+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e212 v212) 2019-12-11T21:59:33.846 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:27:20.776 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:33.886 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:27:20.816 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:33.893 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:27:20.824 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:33.904 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:27:20.834 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:33.910 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:27:20.811 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:33.922 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:27:20.819 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:33.992 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:27:20.921 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:34.002 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:27:20.904 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:34.025 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:34.020+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:34.059 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:34.051+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:34.059 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:34.053+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:34.062 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:27:20.991 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T21:59:34.066 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:34.060+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:34.066 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:34.061+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:34.108 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:34.102+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:34.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:34.436+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:34.549 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:34.544+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:34.762 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:34.755+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:35.024 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:35.017+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:35.062 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:35.050+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:35.062 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:35.053+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:35.117 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:35.111+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:35.118 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:35.113+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:35.169 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:35.164+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:35.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:35.405+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:35.530 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:35.524+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:35.769 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:35.762+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:36.072 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:36.061+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:36.081 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:36.072+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:36.081 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:36.073+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:36.082 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:36.075+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:36.083 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:36.077+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:36.190 INFO:teuthology.orchestra.run.smithi079.stderr:testMetadataOnPost (test.functional.tests.TestFile) ... ok 2019-12-11T21:59:36.190 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:36.185+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:36.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:36.404+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:36.489 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:36.484+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:36.757 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:36.749+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:37.091 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:37.084+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:37.092 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:37.087+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:37.094 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:37.088+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:37.095 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:37.090+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:37.117 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:37.109+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:37.211 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:37.205+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:37.458 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:37.453+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:37.464 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:37.457+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:37.773 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:37.765+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:38.073 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:38.064+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:38.073 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:38.062+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:38.074 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:38.065+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:38.077 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:38.069+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:38.122 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:38.112+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:38.203 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:38.197+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:38.433 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:38.422+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:38.493 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:38.484+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:38.770 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:38.759+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:38.843 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:59:38.837+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e212 v212) 2019-12-11T21:59:39.023 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:39.017+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:39.029 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:39.019+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:39.048 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:39.036+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:39.053 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:39.046+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:39.121 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:39.115+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:39.235 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:39.228+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:39.464 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:39.456+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:39.502 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:39.494+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:39.732 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:39.725+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:40.039 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:40.033+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:40.041 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:40.037+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:40.043 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:40.037+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:40.043 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:40.038+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:40.147 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:40.140+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:40.284 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:40.278+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:40.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:40.499+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:40.536 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:40.531+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:40.759 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:40.751+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:41.070 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:41.064+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:41.070 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:41.062+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:41.071 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:41.066+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:41.071 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:41.064+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:41.180 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:41.173+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:41.246 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:41.234+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:41.479 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:41.469+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:41.528 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:41.523+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:41.792 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:41.786+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:42.080 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:42.073+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:42.080 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:42.075+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:42.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:42.097+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:42.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:42.099+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:42.205 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:42.199+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:42.291 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:42.285+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:42.502 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:42.492+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:42.535 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:42.530+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:42.775 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:42.769+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:43.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:43.096+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:43.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:43.098+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:43.120 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:43.109+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:43.127 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:43.118+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:43.188 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:43.181+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:43.266 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:43.260+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:43.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:43.468+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:43.532 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:43.526+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:43.777 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:43.771+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:43.851 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:59:43.848+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e212 v212) 2019-12-11T21:59:44.089 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:44.083+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:44.090 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:44.084+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:44.133 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:44.129+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:44.135 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:44.131+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:44.234 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:44.228+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:44.239 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:44.232+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:44.490 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:44.483+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:44.524 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:44.519+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:44.820 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:44.813+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:45.133 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:45.124+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:45.133 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:45.128+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:45.134 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:45.108+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:45.134 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:45.114+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:45.262 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:45.254+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:45.278 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:45.273+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:45.479 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:45.472+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:45.515 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:45.508+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:45.817 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:45.810+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:46.104 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:46.096+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:46.105 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:46.099+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:46.155 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:46.149+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:46.155 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:46.151+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:46.249 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:46.244+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:46.301 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:46.291+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:46.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:46.433+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:46.513 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:46.507+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:46.848 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:46.836+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:47.121 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:47.112+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:47.121 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:47.115+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:47.169 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:47.162+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:47.169 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:47.165+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:47.267 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:47.260+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:47.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:47.288+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:47.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:47.466+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:47.488 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:47.481+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:47.825 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:47.817+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:48.090 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:48.082+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:48.090 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:48.084+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:48.180 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:48.173+0000 1694f700 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.885287+0000 front 2019-12-11T21:58:15.772269+0000 (oldest deadline 2019-12-11T21:58:56.480812+0000) 2019-12-11T21:59:48.180 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:48.176+0000 1694f700 -1 osd.3 212 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:48.228 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T21:59:48.216+0000 1694f700 -1 osd.4 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.450676+0000 front 2019-12-11T21:58:14.507503+0000 (oldest deadline 2019-12-11T21:58:59.001523+0000) 2019-12-11T21:59:48.348 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T21:59:48.339+0000 1694f700 -1 osd.2 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:16.080736+0000 front 2019-12-11T21:58:16.064167+0000 (oldest deadline 2019-12-11T21:59:00.327924+0000) 2019-12-11T21:59:48.503 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T21:59:48.495+0000 1694f700 -1 osd.7 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:14.994887+0000 front 2019-12-11T21:58:15.019841+0000 (oldest deadline 2019-12-11T21:59:00.279892+0000) 2019-12-11T21:59:48.517 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T21:59:48.511+0000 1694f700 -1 osd.0 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:12.841008+0000 front 2019-12-11T21:58:12.911231+0000 (oldest deadline 2019-12-11T21:58:56.319433+0000) 2019-12-11T21:59:48.838 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T21:59:48.830+0000 1694f700 -1 osd.6 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T21:58:15.250582+0000 front 2019-12-11T21:58:15.024178+0000 (oldest deadline 2019-12-11T21:58:57.314668+0000) 2019-12-11T21:59:48.925 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T21:59:48.918+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e212 v212) 2019-12-11T21:59:49.097 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:49.089+0000 1694f700 -1 osd.5 212 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T21:58:33.925981+0000 (oldest deadline 2019-12-11T21:59:13.925981+0000) 2019-12-11T21:59:49.100 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T21:59:49.091+0000 1694f700 -1 osd.5 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e210) 2019-12-11T21:59:49.851 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T21:59:49.794+0000 1694f700 -1 osd.3 213 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106166 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5479265296 gen 1] snapc 0=[] ondisk+write+known_if_redirected e209) 2019-12-11T21:59:51.283 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T21:59:51.274+0000 20e25700 -1 RGWWatcher::handle_error cookie 5479265296 err (107) Transport endpoint is not connected 2019-12-11T22:00:17.985 INFO:teuthology.orchestra.run.smithi079.stderr:testMetadataOnPut (test.functional.tests.TestFile) ... ok 2019-12-11T22:01:11.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:11.689+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:12.745 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:12.738+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:13.302 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:13.296+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:13.789 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:13.782+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:14.186 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:14.180+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:14.268 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:14.260+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:14.485 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:14.480+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:14.803 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:14.797+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:15.118 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:15.113+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:15.196 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:15.188+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:15.225 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:15.219+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:15.513 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:15.507+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:15.799 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:15.790+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:16.073 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:16.069+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:16.149 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:16.144+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:16.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:16.188+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:16.559 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:16.554+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:16.851 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:16.844+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:17.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:17.043+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:17.120 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:17.115+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:17.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:17.182+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:17.363 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:17.352+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:17.609 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:17.603+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:17.867 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:17.861+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:18.021 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:18.016+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:18.129 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:18.118+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:18.190 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:18.183+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:18.405 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:18.401+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:18.564 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:18.559+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:18.841 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:18.825+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:19.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:19.043+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:19.098 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:19.093+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:19.191 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:19.185+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:19.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:19.216+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:19.366 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:19.361+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:19.599 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:19.593+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:19.809 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:19.802+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:20.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:20.011+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:20.075 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:20.070+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:20.161 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:20.155+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:20.239 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:20.229+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:20.359 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:20.351+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:20.593 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:20.588+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:20.797 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:20.791+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:21.018 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:21.012+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:21.111 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:21.106+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:21.205 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:21.200+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:21.243 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:21.239+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:21.379 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:21.373+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:21.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:21.583+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:21.803 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:21.798+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:22.052 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:22.046+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:22.079 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:22.073+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:22.167 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:22.161+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:22.255 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:22.251+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:22.342 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:22.335+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:22.613 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:22.598+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:22.825 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:22.820+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:23.060 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:23.049+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:23.063 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:23.058+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:23.198 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:23.192+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:23.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:23.289+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:23.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:23.289+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:23.587 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:23.582+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:23.814 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:23.808+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:24.054 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:24.049+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:24.059 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:24.054+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:24.192 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:24.186+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:24.259 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:24.254+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:24.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:24.290+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:24.568 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:24.563+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:24.774 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:24.768+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:25.020 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:25.015+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:25.027 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:25.022+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:25.243 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:25.238+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:25.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:25.267+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:25.304 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:25.298+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:25.623 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:25.616+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:25.777 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:25.771+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:26.049 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:26.041+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:26.071 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:26.066+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:26.236 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:26.227+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:26.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:26.254+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:26.307 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:26.301+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:26.615 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:26.610+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:26.744 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:26.738+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:27.070 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:27.063+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:27.081 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:27.076+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:27.206 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:27.201+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:27.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:27.265+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:27.336 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:27.331+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:27.595 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:27.590+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:27.745 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:27.740+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:28.063 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:28.057+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:28.090 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:28.085+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:28.179 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:28.172+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:28.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:28.270+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:28.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:28.272+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:28.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:28.317+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:28.637 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:28.631+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:28.797 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:28.790+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:29.079 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:29.069+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:29.104 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:29.093+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:29.222 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:29.214+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:29.282 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:29.276+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:29.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:29.280+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:29.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:29.314+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:29.649 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:29.644+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:29.782 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:29.776+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:30.094 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:30.087+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:30.094 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:30.088+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:30.223 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:30.217+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:30.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:30.264+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:30.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:30.266+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:30.287 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:30.281+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:30.665 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:30.654+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:30.821 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:30.814+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:31.054 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:31.048+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:31.111 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:31.105+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:31.242 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:31.236+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:31.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:31.250+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:31.263 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:31.251+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:31.286 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:31.280+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:31.684 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:31.679+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:31.845 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:31.839+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:32.035 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:32.030+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:32.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:32.104+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:32.206 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:32.200+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:32.308 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:32.302+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:32.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:32.305+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:32.333 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:32.328+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:32.737 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:32.731+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:32.863 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:32.856+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:33.013 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:33.004+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:33.073 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:33.068+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:33.203 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:33.197+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:33.306 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:33.300+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:33.312 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:33.306+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:33.384 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:33.380+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:33.700 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:33.694+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:33.897 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:33.891+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:34.044 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:34.038+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:34.055 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:34.049+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:34.159 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:34.152+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:34.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:34.320+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:34.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:34.321+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:34.363 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:34.356+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:34.730 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:34.725+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:34.917 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:34.911+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:35.086 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:35.074+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:35.086 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:35.076+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:35.205 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:35.195+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:35.305 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:35.298+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:35.305 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:35.299+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:35.396 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:35.390+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:35.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:35.754+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:35.904 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:35.898+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:36.128 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:36.122+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:36.133 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:36.127+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:36.216 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:36.210+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:36.314 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:36.307+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:36.314 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:36.308+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:36.413 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:36.406+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:36.729 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:36.722+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:36.899 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:36.888+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:37.084 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:37.078+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:37.156 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:37.149+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:37.200 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:37.194+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:37.274 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:37.267+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:37.274 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:37.269+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:37.453 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:37.446+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:37.726 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:37.717+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:37.922 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:37.916+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:38.049 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:38.043+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:38.110 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:38.105+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:38.221 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:38.215+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:38.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:38.309+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:38.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:38.313+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:38.475 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:38.468+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:38.773 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:38.767+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:38.953 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:38.946+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:39.031 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:39.026+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:39.138 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:39.131+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:39.218 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:39.212+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:39.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:39.280+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:39.292 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:39.283+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:39.511 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:39.506+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:39.732 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:39.726+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:39.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:39.993+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:40.047 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:40.039+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:40.184 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:40.177+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:40.188 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:40.180+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:40.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:40.282+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:40.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:40.284+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:40.516 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:40.510+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:40.734 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:40.729+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:41.000 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:40.993+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:41.038 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:41.033+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:41.168 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:41.162+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:41.182 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:41.177+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:41.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:41.329+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:41.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:41.331+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:41.484 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:41.479+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:41.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:41.781+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:42.046 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:42.042+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:42.049 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:42.044+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:42.169 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:42.159+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:42.169 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:42.163+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:42.301 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:42.292+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:42.304 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:42.299+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:42.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:42.473+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:42.849 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:42.834+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:43.007 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:43.000+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:43.070 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:43.064+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:43.157 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:43.151+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:43.214 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:43.208+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:43.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:43.328+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:43.337 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:43.330+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:43.521 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:43.515+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:43.865 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:43.858+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:44.043 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:44.036+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:44.067 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:44.063+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:44.186 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:44.180+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:44.238 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:44.231+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:44.317 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:01:44.310+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e217 v217) 2019-12-11T22:01:44.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:44.357+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:44.366 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:44.361+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:44.529 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:44.523+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:44.832 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:44.825+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:45.037 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:45.031+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:45.064 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:45.057+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:45.181 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:45.164+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:45.260 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:45.254+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:45.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:45.404+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:45.412 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:45.405+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:45.498 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:45.493+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:45.795 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:45.789+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:46.075 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:46.069+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:46.082 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:46.077+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:46.147 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:46.128+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:46.234 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:46.228+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:46.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:46.374+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:46.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:46.380+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:46.501 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:46.496+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:46.762 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:46.756+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:47.057 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:47.053+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:47.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:47.081+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:47.138 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:47.131+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:47.256 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:47.251+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:47.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:47.333+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:47.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:47.334+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:47.501 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:47.495+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:47.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:47.798+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:48.065 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:48.060+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:48.106 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:48.098+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:48.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:48.119+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:48.245 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:48.240+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:48.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:48.306+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:48.314 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:48.308+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:48.552 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:01:48.547+0000 1694f700 -1 osd.2 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:31.987191+0000 front 2019-12-11T22:00:31.577459+0000 (oldest deadline 2019-12-11T22:01:16.728479+0000) 2019-12-11T22:01:48.760 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:01:48.755+0000 1694f700 -1 osd.3 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.964947+0000 front 2019-12-11T22:00:30.524393+0000 (oldest deadline 2019-12-11T22:01:14.398949+0000) 2019-12-11T22:01:49.057 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:01:49.052+0000 1694f700 -1 osd.0 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.939427+0000 front 2019-12-11T22:00:28.999303+0000 (oldest deadline 2019-12-11T22:01:14.007971+0000) 2019-12-11T22:01:49.066 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:01:49.060+0000 1694f700 -1 osd.6 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:32.998443+0000 front 2019-12-11T22:00:28.998779+0000 (oldest deadline 2019-12-11T22:01:12.920221+0000) 2019-12-11T22:01:49.118 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:01:49.111+0000 1694f700 -1 osd.4 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:26.876925+0000 front 2019-12-11T22:00:31.981621+0000 (oldest deadline 2019-12-11T22:01:10.881961+0000) 2019-12-11T22:01:49.250 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:01:49.245+0000 1694f700 -1 osd.1 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:29.196984+0000 front 2019-12-11T22:00:29.138537+0000 (oldest deadline 2019-12-11T22:01:14.426360+0000) 2019-12-11T22:01:49.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:49.344+0000 1694f700 -1 osd.7 217 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:00:28.516102+0000 front 2019-12-11T22:00:27.430539+0000 (oldest deadline 2019-12-11T22:01:27.915816+0000) 2019-12-11T22:01:49.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:49.346+0000 1694f700 -1 osd.7 217 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e217) 2019-12-11T22:01:49.375 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:01:49.371+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e217 v217) 2019-12-11T22:01:50.694 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:50.665+0000 1694f700 -1 osd.7 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:51.696 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:51.666+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:52.644 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:52.639+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:53.650 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:53.645+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:54.705 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:54.698+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:55.679 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:55.674+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:56.730 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:56.725+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:57.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:57.737+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:58.794 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:58.788+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:01:59.835 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:01:59.829+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:00.828 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:00.823+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:01.826 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:01.820+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:02.840 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:02.834+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:03.841 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:03.835+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:04.888 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:04.882+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:05.865 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:05.859+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:06.825 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:06.819+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:07.870 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:07.864+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:08.887 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:08.879+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:09.930 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:09.923+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:10.892 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:10.886+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:11.928 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:11.923+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:12.904 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:12.898+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:13.874 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:13.869+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:14.613 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:30:01.543 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.624 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:30:01.554 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.638 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:30:01.540 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.645 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:30:01.576 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.649 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:30:01.550 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.651 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:30:01.581 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.838 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:30:01.769 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.840 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:30:01.741 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:14.895 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:14.890+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:14.896 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:30:01.826 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:15.852 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:15.846+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:16.892 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:16.887+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:17.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:17.929+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:18.918 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:18.913+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:19.916 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:19.912+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:20.899 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:20.894+0000 1694f700 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:21.901 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:21.868+0000 1694f700 -1 osd.7 220 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:22.832 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:22.828+0000 1694f700 -1 osd.7 220 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:23.885 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:23.881+0000 1694f700 -1 osd.7 220 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:24.915 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:24.910+0000 1694f700 -1 osd.7 220 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e217) 2019-12-11T22:02:27.257 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:02:27.247+0000 34f8d700 -1 osd.5 222 build_incremental_map_msg missing incremental map 222 2019-12-11T22:02:27.400 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:02:27.342+0000 33f8b700 -1 osd.5 222 build_incremental_map_msg missing incremental map 222 2019-12-11T22:02:27.509 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:02:27.484+0000 3578e700 -1 osd.5 222 build_incremental_map_msg missing incremental map 222 2019-12-11T22:02:29.183 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:02:29.128+0000 30f85700 -1 osd.5 223 build_incremental_map_msg missing incremental map 222 2019-12-11T22:02:42.046 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:42.040+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:43.011 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:43.007+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:43.991 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:43.986+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:45.021 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:45.018+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:46.042 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:46.038+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:47.021 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:47.016+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:47.211 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:47.205+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:47.382 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:47.375+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:48.024 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:48.019+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:48.053 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:48.047+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:48.090 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:48.050+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:48.220 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:48.214+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:48.354 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:48.347+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:48.965 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:48.957+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:49.042 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:49.036+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:49.104 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:49.098+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:49.105 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:49.100+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:49.191 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:49.185+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:49.322 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:49.315+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:49.850 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:49.843+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:49.946 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:49.939+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:50.087 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:50.082+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:50.128 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:50.122+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:50.128 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:50.124+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:50.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:50.212+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:50.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:50.215+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:50.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:50.352+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:50.908 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:50.897+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:50.990 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:50.982+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:51.061 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:51.054+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:51.139 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:51.133+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:51.140 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:51.135+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:51.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:51.261+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:51.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:51.265+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:51.356 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:51.351+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:51.935 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:51.927+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:52.004 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:51.996+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:52.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:52.103+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:52.172 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:52.163+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:52.172 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:52.166+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:52.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:52.315+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:52.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:52.317+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:52.366 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:52.359+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:52.971 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:52.963+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:52.996 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:52.988+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:53.111 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:53.098+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:53.175 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:53.170+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:53.178 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:53.173+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:53.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:53.335+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:53.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:53.341+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:53.362 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:53.351+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:53.998 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:53.991+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:54.014 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:54.007+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:54.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:54.087+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:54.199 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:54.194+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:54.202 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:54.198+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:54.314 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:54.306+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:54.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:54.309+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:54.346 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:54.340+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:54.808 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:30:41.739 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.814 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:30:41.745 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.840 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:30:41.771 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.851 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:30:41.753 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.873 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:30:41.774 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.878 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:30:41.808 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.917 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:30:41.847 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.951 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:30:41.881 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:54.956 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:30:41.857 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:02:55.051 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:55.044+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:55.059 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:55.053+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:55.065 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:55.058+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:55.250 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:55.245+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:55.251 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:55.246+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:55.293 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:55.285+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:55.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:55.288+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:55.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:55.322+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:56.023 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:56.018+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:56.029 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:56.022+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:56.075 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:56.068+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:56.231 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:56.227+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:56.232 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:56.229+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:56.261 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:56.254+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:56.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:56.256+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:56.322 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:56.316+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:56.983 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:56.977+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:57.064 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:57.056+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:57.112 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:57.105+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:57.232 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:57.225+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:57.233 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:57.228+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:57.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:57.292+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:57.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:57.296+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:57.339 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:57.334+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:57.982 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:57.976+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:58.076 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:58.068+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:58.081 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:58.072+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:58.198 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:58.194+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:58.199 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:58.195+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:58.276 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:58.267+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:58.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:58.270+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:58.315 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:58.310+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:58.944 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:58.933+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:02:59.049 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:02:59.033+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:02:59.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:02:59.057+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:02:59.163 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:59.156+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:02:59.163 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:02:59.158+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:02:59.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:59.279+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:02:59.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:02:59.284+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:02:59.324 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:02:59.319+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:02:59.869 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:02:59.857+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:02:59.939 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:02:59.935+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:00.011 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:00.005+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:00.034 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:00.026+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:00.160 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:00.155+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:00.161 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:00.156+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:00.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:00.278+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:00.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:00.281+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:00.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:00.329+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:00.874 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:00.868+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:00.967 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:00.961+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:01.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:01.030+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:01.039 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:01.031+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:01.206 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:01.200+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:01.206 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:01.202+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:01.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:01.324+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:01.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:01.326+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:01.366 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:01.360+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:01.920 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:01.914+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:01.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:01.922+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:02.039 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:02.033+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:02.089 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:02.083+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:02.215 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:02.204+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:02.215 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:02.207+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:02.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:02.346+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:02.353 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:02.347+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:02.420 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:02.412+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:02.924 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:02.917+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:02.975 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:02.970+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:03.001 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:02.993+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:03.060 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:03.051+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:03.225 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:03.219+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:03.226 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:03.220+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:03.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:03.306+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:03.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:03.311+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:03.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:03.426+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:03.912 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:03.905+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:03.947 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:03.939+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:03.948 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:03.942+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:04.023 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:04.015+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:04.040 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:04.033+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:04.181 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:04.175+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:04.185 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:04.179+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:04.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:04.337+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:04.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:04.340+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:04.387 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:04.382+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:04.924 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:04.905+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:04.924 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:04.908+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:04.925 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:04.910+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:04.996 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:04.986+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:05.048 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:05.041+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:05.223 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:05.204+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:05.223 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:05.208+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:05.309 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:05.299+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:05.309 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:05.302+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:05.358 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:05.351+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:05.872 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:05.862+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:05.873 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:05.867+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:05.967 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:05.959+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:05.976 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:05.968+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:06.020 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:06.012+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:06.223 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:06.216+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:06.223 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:06.217+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:06.300 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:06.292+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:06.301 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:06.295+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:06.362 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:06.357+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:06.837 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:06.826+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:06.838 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:06.829+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:06.938 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:06.931+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:07.005 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:06.998+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:07.073 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:07.065+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:07.258 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:07.252+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:07.261 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:07.255+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:07.346 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:07.338+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:07.347 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:07.341+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:07.398 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:07.393+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:07.808 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:07.800+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:07.809 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:07.803+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:07.994 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:07.986+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:08.026 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:08.019+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:08.085 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:08.077+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:08.219 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:08.210+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:08.220 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:08.214+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:08.349 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:08.341+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:08.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:08.344+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:08.369 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:08.362+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:08.845 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:08.837+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:08.846 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:08.840+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:09.031 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:09.026+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:09.042 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:09.035+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:09.042 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:09.033+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:09.233 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:09.224+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:09.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:09.229+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:09.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:09.354+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:09.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:09.357+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:09.420 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:09.414+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:09.883 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:09.875+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:09.884 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:09.878+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:10.017 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:10.012+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:10.046 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:10.040+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:10.093 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:10.076+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:10.190 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:10.183+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:10.191 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:10.187+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:10.394 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:10.389+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:10.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:10.397+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:10.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:10.399+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:10.913 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:10.907+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:10.915 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:10.910+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:10.987 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:10.981+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:11.062 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:11.053+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:11.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:11.129+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:11.200 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:11.191+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:11.200 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:11.194+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:11.380 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:11.374+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:11.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:11.396+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:11.406 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:11.401+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:11.940 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:11.931+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:11.941 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:11.935+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:12.036 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:12.031+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:12.082 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:12.075+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:12.165 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:12.158+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:12.237 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:12.231+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:12.237 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:12.233+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:12.431 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:12.425+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:12.447 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:12.439+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:12.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:12.442+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:12.907 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:12.898+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:12.908 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:12.901+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:13.048 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:13.041+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:13.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:13.050+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:13.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:13.189+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:13.212 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:13.205+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:13.212 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:13.208+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:13.407 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:13.399+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:13.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:13.402+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:13.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:13.460+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:13.949 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:13.941+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:13.950 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:13.944+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:14.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:14.012+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:14.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:14.030+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:14.184 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:14.177+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:14.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:14.228+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:14.235 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:14.230+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:14.450 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:14.443+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:14.451 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:14.445+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:14.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:14.464+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:14.937 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:14.930+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:14.938 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:14.933+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:15.029 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:15.023+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:15.050 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:15.046+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:15.149 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:15.142+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:15.202 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:15.195+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:15.205 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:15.200+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:15.499 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:15.492+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:15.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:15.495+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:15.502 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:15.498+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:15.977 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:15.970+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:15.981 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:15.977+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:16.040 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:16.034+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:16.041 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:16.036+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:16.123 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:16.117+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:16.224 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:16.219+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:16.227 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:16.223+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:16.458 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:16.452+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:16.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:16.501+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:16.509 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:16.505+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:17.024 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:17.015+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:17.030 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:17.023+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:17.031 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:17.026+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:17.063 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:17.057+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:17.112 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:17.100+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:17.236 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:17.230+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:17.237 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:17.233+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:17.478 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:17.473+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:17.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:17.493+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:17.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:17.495+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:17.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:17.970+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:17.998 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:17.993+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:17.999 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:17.995+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:18.037 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:18.031+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:18.102 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:18.095+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:18.208 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:18.203+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:18.209 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:18.205+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:18.460 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:18.454+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:18.461 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:18.456+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:18.512 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:18.507+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:19.015 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:19.009+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:19.019 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:19.014+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:19.027 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:19.021+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:19.027 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:19.021+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:19.152 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:19.146+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:19.235 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:19.230+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:19.237 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:19.231+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:19.454 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:19.447+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:19.457 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:19.452+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:19.513 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:19.506+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:19.834 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:03:19.829+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 40sec e223 v223) 2019-12-11T22:03:20.009 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:20.004+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:20.026 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:20.016+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:20.026 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:20.021+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:20.078 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:20.072+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:20.203 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:20.196+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:20.205 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:20.199+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:20.206 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:20.201+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:20.508 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:20.500+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:20.508 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:20.503+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:20.567 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:20.560+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:21.031 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:21.026+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:21.037 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:21.030+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:21.038 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:21.033+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:21.049 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:21.043+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:21.169 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:21.163+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:21.170 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:21.165+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:21.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:21.242+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:21.536 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:21.529+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:21.537 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:21.531+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:21.591 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:21.580+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:22.020 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:22.012+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:22.044 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:22.035+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:22.051 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:22.038+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:22.063 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:22.058+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:22.200 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:22.195+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:22.202 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:22.198+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:22.261 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:22.255+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:22.571 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:22.564+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:22.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:22.565+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:22.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:22.567+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:23.010 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:23.004+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:23.034 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:23.029+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:23.035 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:23.029+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:23.039 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:23.034+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:23.243 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:23.236+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:23.248 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:23.241+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:23.249 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:23.244+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:23.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:23.577+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:23.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:23.579+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:23.607 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:23.600+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:23.988 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:23.981+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:24.054 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:24.045+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:24.057 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:24.049+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:24.060 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:24.052+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:24.207 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:24.200+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:24.256 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:24.251+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:24.261 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:24.254+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:24.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:24.568+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:24.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:24.618+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:24.626 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:24.620+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:24.892 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:03:24.886+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 40sec e223 v223) 2019-12-11T22:03:25.016 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:25.008+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:25.050 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:25.042+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:25.053 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:25.047+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:25.068 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:25.063+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:25.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:25.166+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:25.287 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:25.281+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:25.290 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:25.283+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:25.546 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:25.539+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:25.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:25.581+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:25.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:25.583+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:26.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:26.031+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:26.048 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:26.043+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:26.083 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:26.076+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:26.086 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:26.081+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:26.165 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:26.157+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:26.314 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:26.307+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:26.332 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:26.327+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:26.516 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:26.511+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:26.565 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:26.558+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:26.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:26.563+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:27.003 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:26.996+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:27.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:27.009+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:27.082 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:27.073+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:27.083 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:27.076+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:27.187 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:27.178+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:27.302 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:27.298+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:27.303 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:27.300+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:27.531 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:27.526+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:27.540 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:27.533+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:27.543 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:27.538+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:27.989 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:27.980+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:28.017 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:28.010+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:28.041 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:28.033+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:28.044 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:28.039+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:28.200 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:28.192+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:28.348 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:28.344+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:28.353 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:28.347+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:28.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:28.493+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:28.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:28.498+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:28.547 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:28.540+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:29.037 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:29.031+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:29.042 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:29.034+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:29.091 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:29.083+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:29.092 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:29.086+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:29.161 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:29.154+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:29.349 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:29.344+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:29.350 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:29.345+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:29.528 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:29.520+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:29.529 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:29.523+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:29.584 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:29.579+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:29.915 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:03:29.907+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 40sec e223 v223) 2019-12-11T22:03:30.043 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:30.037+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:30.072 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:30.067+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:30.077 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:30.072+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:30.079 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:30.074+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:30.148 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:30.141+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:30.331 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:30.325+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:30.332 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:30.327+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:30.520 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:30.512+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:30.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:30.515+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:30.568 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:30.562+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:31.054 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:31.047+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:31.099 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:31.093+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:31.133 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:31.124+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:31.135 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:31.127+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:31.176 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:31.169+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:31.334 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:31.326+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:31.335 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:31.329+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:31.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:31.526+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:31.536 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:31.531+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:31.575 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:31.569+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:32.048 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:32.040+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:32.096 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:32.091+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:32.097 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:32.091+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:32.100 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:32.094+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:32.146 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:32.137+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:32.369 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:32.364+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:32.370 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:32.365+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:32.494 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:32.488+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:32.495 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:32.489+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:32.577 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:32.572+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:33.054 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:33.046+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:33.055 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:33.050+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:33.058 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:33.050+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:33.059 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:33.053+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:33.144 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:33.136+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:33.356 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:33.351+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:33.359 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:33.355+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:33.549 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:33.537+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:33.549 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:33.539+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:33.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:33.535+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:34.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:34.032+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:34.057 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:34.050+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:34.057 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:34.052+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:34.107 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:03:34.102+0000 1694f700 -1 osd.2 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:02.248567+0000 front 2019-12-11T22:02:02.177201+0000 (oldest deadline 2019-12-11T22:02:47.455549+0000) 2019-12-11T22:03:34.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:03:34.130+0000 1694f700 -1 osd.6 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.395844+0000 front 2019-12-11T22:02:05.395087+0000 (oldest deadline 2019-12-11T22:02:48.872528+0000) 2019-12-11T22:03:34.313 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:34.307+0000 1694f700 -1 osd.3 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.715993+0000 front 2019-12-11T22:02:05.655649+0000 (oldest deadline 2019-12-11T22:02:47.946434+0000) 2019-12-11T22:03:34.316 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:34.311+0000 1694f700 -1 osd.3 223 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:34.517 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:03:34.513+0000 1694f700 -1 osd.0 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:07.175897+0000 front 2019-12-11T22:02:07.004356+0000 (oldest deadline 2019-12-11T22:02:47.244088+0000) 2019-12-11T22:03:34.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:34.526+0000 1694f700 -1 osd.7 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:05.746077+0000 front 2019-12-11T22:02:05.741437+0000 (oldest deadline 2019-12-11T22:02:49.828266+0000) 2019-12-11T22:03:34.537 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:03:34.531+0000 1694f700 -1 osd.7 223 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2.efbd0d20 (undecoded) ondisk+retry+write+known_if_redirected e219) 2019-12-11T22:03:34.968 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:03:34.958+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 40sec e223 v223) 2019-12-11T22:03:35.026 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:03:35.019+0000 1694f700 -1 osd.4 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:02:04.271479+0000 front 2019-12-11T22:02:04.363543+0000 (oldest deadline 2019-12-11T22:02:49.517896+0000) 2019-12-11T22:03:35.058 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:35.042+0000 1694f700 -1 osd.5 223 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:02:23.047145+0000 (oldest deadline 2019-12-11T22:03:03.047145+0000) 2019-12-11T22:03:35.062 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:03:35.057+0000 1694f700 -1 osd.5 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e221) 2019-12-11T22:03:35.806 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:35.802+0000 1694f700 -1 osd.3 224 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:106760 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e219) 2019-12-11T22:03:36.965 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:03:36.853+0000 1694f700 -1 osd.3 225 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:106820 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 820376144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e223) 2019-12-11T22:03:37.289 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:03:37.210+0000 20e25700 -1 RGWWatcher::handle_error cookie 820376144 err (107) Transport endpoint is not connected 2019-12-11T22:04:31.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:31.296+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:32.314 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:32.309+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:33.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:33.278+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:34.247 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:34.241+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:35.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:35.273+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:36.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:36.320+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:37.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:37.367+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:38.406 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:38.400+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:39.361 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:39.355+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:40.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:40.345+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:41.356 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:41.351+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:42.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:42.380+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:43.423 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:43.416+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:44.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:44.460+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:45.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:45.494+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:46.527 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:46.521+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:47.484 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:47.478+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:48.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:48.494+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:49.535 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:49.529+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:50.527 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:50.520+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:51.494 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:51.489+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:52.482 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:52.476+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:53.467 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:53.462+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:54.468 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:54.462+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:55.505 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:55.498+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:56.530 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:56.521+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:57.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:57.505+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:58.469 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:58.463+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:04:59.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:04:59.494+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:00.487 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:00.482+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:00.703 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:00.697+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:01.304 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:01.299+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:01.454 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:01.449+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:01.681 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:01.676+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:02.314 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:02.308+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:02.419 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:02.414+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:02.485 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:02.480+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:02.692 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:02.686+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:03.268 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:03.264+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:03.414 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:03.407+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:03.415 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:03.409+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:03.502 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:03.495+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:03.503 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:03.497+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:03.660 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:03.654+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:04.322 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:04.316+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:04.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:04.400+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:04.431 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:04.425+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:04.515 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:04.509+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:04.518 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:04.513+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:04.636 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:04.631+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:05.286 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:05.281+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:05.398 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:05.389+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:05.413 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:05.409+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:05.448 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:05.441+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:05.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:05.552+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:05.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:05.554+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:05.651 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:05.647+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:05.766 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:05.760+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:06.285 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:06.279+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:06.356 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:06.349+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:06.380 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:06.374+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:06.503 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:06.493+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:06.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:06.569+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:06.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:06.570+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:06.672 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:06.668+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:06.784 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:06.778+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:07.291 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:07.286+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:07.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:07.402+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:07.408 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:07.403+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:07.488 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:07.482+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:07.596 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:07.585+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:07.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:07.592+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:07.658 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:07.651+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:07.749 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:07.745+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:08.289 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:08.285+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:08.406 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:08.401+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:08.421 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:08.414+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:08.517 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:08.512+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:08.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:08.637+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:08.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:08.638+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:08.672 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:08.656+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:08.756 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:08.749+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:09.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:09.324+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:09.397 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:09.393+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:09.400 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:09.395+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:09.504 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:09.500+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:09.621 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:09.615+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:09.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:09.661+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:09.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:09.663+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:09.779 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:09.774+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:10.351 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:10.345+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:10.392 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:10.387+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:10.443 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:10.435+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:10.540 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:10.535+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:10.609 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:10.604+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:10.635 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:10.626+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:10.635 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:10.629+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:10.797 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:10.791+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:11.345 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:11.340+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:11.430 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:11.425+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:11.471 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:11.464+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:11.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:11.521+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:11.594 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:11.589+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:11.685 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:11.674+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:11.688 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:11.684+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:11.818 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:11.812+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:12.328 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:12.323+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:12.397 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:12.387+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:12.482 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:12.478+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:12.501 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:12.495+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:12.567 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:12.561+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:12.731 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:12.725+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:12.731 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:12.727+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:12.863 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:12.856+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:13.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:13.323+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:13.416 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:13.411+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:13.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:13.472+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:13.523 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:13.518+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:13.527 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:13.520+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:13.705 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:13.700+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:13.707 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:13.703+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:13.864 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:13.859+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:14.363 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:14.357+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:14.385 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:14.378+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:14.460 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:14.454+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:14.486 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:14.481+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:14.537 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:14.530+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:14.761 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:14.754+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:14.761 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:14.756+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:14.849 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:14.844+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:15.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:15.361+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:15.383 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:15.378+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:15.437 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:15.431+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:15.488 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:15.482+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:15.529 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:15.523+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:15.773 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:15.767+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:15.774 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:15.767+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:15.891 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:15.884+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:16.377 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:16.370+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:16.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:16.374+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:16.399 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:16.391+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:16.495 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:16.488+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:16.559 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:16.552+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:16.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:16.792+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:16.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:16.794+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:16.893 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:16.888+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:17.399 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:17.393+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:17.422 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:17.417+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:17.429 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:17.422+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:17.497 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:17.475+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:17.578 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:17.568+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:17.758 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:17.751+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:17.759 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:17.752+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:17.865 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:17.858+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:18.391 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:18.384+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:18.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:18.390+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:18.453 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:18.446+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:18.466 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:18.460+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:18.529 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:18.522+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:18.804 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:18.797+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:18.805 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:18.799+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:18.877 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:18.871+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:19.369 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:19.363+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:19.440 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:19.432+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:19.478 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:19.472+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:19.506 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:19.500+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:19.578 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:19.572+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:19.769 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:19.762+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:19.770 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:19.763+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:19.862 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:19.855+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:20.396 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:20.390+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:20.403 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:20.398+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:20.496 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:20.492+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:20.517 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:20.510+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:20.596 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:20.591+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:20.814 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:20.807+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:20.815 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:20.809+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:20.841 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:20.835+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:21.393 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:21.388+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:21.431 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:21.425+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:21.537 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:21.531+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:21.547 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:21.541+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:21.577 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:21.571+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:21.800 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:21.793+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:21.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:21.845+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:21.854 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:21.847+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:22.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:22.356+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:22.471 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:22.463+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:22.510 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:22.505+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:22.550 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:22.543+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:22.570 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:22.566+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:22.765 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:22.759+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:22.873 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:22.867+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:22.876 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:22.871+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:23.396 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:23.391+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:23.423 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:23.418+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:23.470 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:23.462+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:23.546 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:23.539+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:23.597 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:23.590+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:23.729 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:23.723+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:23.880 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:23.874+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:23.880 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:23.875+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:24.404 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:24.399+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:24.425 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:24.421+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:24.460 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:24.455+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:24.515 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:24.510+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:24.629 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:24.622+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:24.773 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:24.767+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:24.836 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:24.829+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:24.836 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:24.831+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:25.417 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:25.412+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:25.429 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:25.424+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:25.467 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:25.462+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:25.509 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:25.503+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:25.595 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:25.588+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:25.824 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:25.818+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:25.833 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:25.824+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:25.835 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:25.826+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:26.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:26.402+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:26.437 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:26.432+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:26.446 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:26.441+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:26.550 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:26.543+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:26.557 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:26.552+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:26.860 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:26.854+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:26.861 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:26.856+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:26.870 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:26.863+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:27.364 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:27.357+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:27.403 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:27.398+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:27.486 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:27.481+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:27.540 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:27.533+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:27.594 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:27.587+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:27.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:27.879+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:27.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:27.880+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:27.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:27.898+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:28.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:28.313+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:28.423 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:28.418+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:28.496 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:28.489+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:28.507 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:28.500+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:28.617 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:28.612+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:28.870 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:28.862+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:28.870 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:28.864+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:28.880 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:28.873+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:29.287 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:29.281+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:29.441 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:29.436+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:29.493 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:29.488+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:29.533 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:29.526+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:29.594 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:29.589+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:29.891 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:29.884+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:29.891 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:29.885+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:29.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:29.897+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:30.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:30.313+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:30.410 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:30.405+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:30.498 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:30.494+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:30.526 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:30.518+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:30.615 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:30.609+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:30.862 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:30.855+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:30.900 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:30.892+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:30.901 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:30.894+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:31.278 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:31.273+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:31.451 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:31.446+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:31.517 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:31.512+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:31.548 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:31.542+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:31.576 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:31.571+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:31.883 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:31.876+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:31.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:31.927+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:31.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:31.929+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:32.330 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:32.325+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:32.408 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:32.403+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:32.506 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:32.501+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:32.555 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:32.549+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:32.584 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:32.579+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:32.875 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:32.868+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:32.948 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:32.942+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:32.951 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:32.946+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:33.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:33.355+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:33.375 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:33.367+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:33.516 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:33.510+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:33.569 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:33.562+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:33.635 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:33.630+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:33.917 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:33.911+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:33.948 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:33.941+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:33.948 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:33.943+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:34.350 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:34.344+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:34.406 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:34.401+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:34.537 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:34.531+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:34.567 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:34.562+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:34.642 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:34.637+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:34.881 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:34.876+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:34.983 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:34.973+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:34.983 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:34.974+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:35.357 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:05:35.348+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 44sec e228 v228) 2019-12-11T22:05:35.379 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:35.374+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:35.410 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:35.405+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:35.540 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:35.533+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:35.585 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:35.580+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:35.673 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:35.665+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:35.918 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:35.905+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:35.938 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:35.931+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:35.940 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:35.934+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:36.384 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:36.379+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:36.423 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:36.417+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:36.575 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:36.567+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:36.595 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:36.589+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:36.645 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:36.640+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:36.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:36.925+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:36.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:36.928+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:36.964 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:36.957+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:37.407 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:37.402+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:37.448 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:37.443+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:37.563 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:37.557+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:37.570 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:37.562+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:37.651 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:37.645+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:37.945 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:37.938+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:37.977 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:37.970+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:37.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:37.972+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:38.375 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:38.370+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:38.460 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:38.455+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:38.540 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:38.535+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:38.596 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:38.590+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:38.642 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:38.637+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:38.901 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:38.896+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:38.947 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:38.939+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:38.947 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:38.940+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:39.403 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:39.398+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:39.509 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:39.504+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:39.546 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:39.542+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:39.554 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:39.548+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:39.619 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:05:39.614+0000 1694f700 -1 osd.1 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.051631+0000 front 2019-12-11T22:04:17.954395+0000 (oldest deadline 2019-12-11T22:05:00.147390+0000) 2019-12-11T22:05:39.871 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:05:39.858+0000 1694f700 -1 osd.6 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.216097+0000 front 2019-12-11T22:04:19.218880+0000 (oldest deadline 2019-12-11T22:05:05.070860+0000) 2019-12-11T22:05:39.915 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:39.909+0000 1694f700 -1 osd.7 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.858878+0000 front 2019-12-11T22:04:17.860882+0000 (oldest deadline 2019-12-11T22:05:02.993650+0000) 2019-12-11T22:05:39.915 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:39.910+0000 1694f700 -1 osd.7 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:40.420 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:05:40.412+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 44sec e228 v228) 2019-12-11T22:05:40.423 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:05:40.419+0000 1694f700 -1 osd.3 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:19.017140+0000 front 2019-12-11T22:04:19.065143+0000 (oldest deadline 2019-12-11T22:05:04.656817+0000) 2019-12-11T22:05:40.462 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:05:40.458+0000 1694f700 -1 osd.2 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:18.060018+0000 front 2019-12-11T22:04:18.001585+0000 (oldest deadline 2019-12-11T22:05:01.971644+0000) 2019-12-11T22:05:40.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:05:40.535+0000 1694f700 -1 osd.0 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:15.373970+0000 front 2019-12-11T22:04:15.372494+0000 (oldest deadline 2019-12-11T22:05:01.248865+0000) 2019-12-11T22:05:40.573 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:05:40.564+0000 1694f700 -1 osd.4 228 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:04:17.838094+0000 front 2019-12-11T22:04:17.220691+0000 (oldest deadline 2019-12-11T22:05:02.552385+0000) 2019-12-11T22:05:41.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:41.429+0000 1694f700 -1 osd.7 229 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:42.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:42.516+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:43.522 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:43.516+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:44.555 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:44.549+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:45.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:45.528+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:46.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:46.576+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:47.585 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:47.576+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:48.551 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:48.545+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:49.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:49.553+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:50.522 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:50.516+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:51.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:51.527+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:52.517 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:52.512+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:53.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:53.506+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:54.549 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:54.543+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:54.952 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:33:41.883 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:54.963 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:33:41.894 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:54.975 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:33:41.904 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:54.977 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:33:41.908 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:54.984 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:33:41.886 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:55.003 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:33:41.905 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:55.150 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:33:42.081 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:55.187 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:33:42.117 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:55.193 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:33:42.094 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:05:55.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:55.557+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:56.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:56.526+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:57.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:57.558+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:58.580 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:58.574+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:05:59.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:05:59.611+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:00.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:00.566+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:01.570 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:01.564+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:02.599 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:02.593+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:03.580 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:03.574+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:04.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:04.625+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:05.683 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:05.679+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:06.666 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:06.661+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:07.664 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:07.660+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:08.686 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:08.681+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:09.689 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:09.684+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:10.667 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:10.663+0000 1694f700 -1 osd.7 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:11.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:11.622+0000 1694f700 -1 osd.7 231 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:12.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:12.619+0000 1694f700 -1 osd.7 231 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:13.665 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:13.660+0000 1694f700 -1 osd.7 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:14.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:14.654+0000 1694f700 -1 osd.7 233 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:15.686 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:15.675+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:15.729 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:15.662+0000 30f85700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:15.785 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:15.705+0000 30f85700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:15.855 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:15.719+0000 30f85700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:15.877 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:15.797+0000 35f8f700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:15.984 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:15.976+0000 37792700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:16.258 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:16.240+0000 30f85700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:16.366 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:16.336+0000 31f87700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:16.642 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:16.636+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:16.686 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:16.506+0000 34f8d700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:17.086 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:17.063+0000 3478c700 -1 osd.5 234 build_incremental_map_msg missing incremental map 234 2019-12-11T22:06:17.653 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:17.648+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:18.729 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:18.680+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:19.706 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:19.701+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:20.690 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:20.680+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:21.681 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:21.675+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:22.687 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:22.682+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:23.713 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:23.707+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:24.721 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:24.715+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:25.741 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:25.735+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:26.792 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:26.782+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:27.841 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:27.827+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:28.832 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:28.826+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:29.852 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:29.845+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:30.834 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:30.822+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:31.823 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:31.817+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:32.802 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:32.797+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:33.765 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:33.757+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:34.758 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:34.751+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:35.798 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:35.791+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:36.768 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:36.761+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:37.751 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:37.745+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:38.730 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:38.724+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:39.155 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:39.148+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:39.186 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:39.181+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:39.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:39.187+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:39.685 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:39.679+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:40.168 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:40.163+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:40.220 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:40.208+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:40.221 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:40.210+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:40.556 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:40.548+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:40.674 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:40.661+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:40.689 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:40.683+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:41.176 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:41.172+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:41.185 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:41.178+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:41.243 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:41.235+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:41.537 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:41.528+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:41.693 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:41.685+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:41.714 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:41.708+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:42.160 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:42.153+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:42.168 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:42.161+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:42.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:42.276+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:42.491 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:42.484+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:42.702 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:42.696+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:42.719 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:42.709+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:42.719 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:42.712+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:43.178 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:43.171+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:43.180 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:43.175+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:43.220 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:43.208+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:43.333 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:43.327+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:43.531 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:43.525+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:43.701 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:43.693+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:43.701 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:43.696+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:43.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:43.727+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:44.201 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:44.190+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:44.202 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:44.197+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:44.248 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:44.240+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:44.343 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:44.337+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:44.576 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:44.569+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:44.693 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:44.687+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:44.754 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:44.746+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:44.754 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:44.748+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:45.251 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:45.244+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:45.252 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:45.246+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:45.304 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:45.292+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:45.393 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:45.387+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:45.557 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:45.550+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:45.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:45.735+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:45.744 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:45.737+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:45.745 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:45.740+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:46.238 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:46.231+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:46.238 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:46.233+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:46.325 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:46.313+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:46.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:46.347+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:46.571 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:46.564+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:46.723 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:46.717+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:46.796 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:46.787+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:46.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:46.787+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:46.801 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:46.790+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:47.216 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:47.209+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:47.217 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:47.211+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:47.321 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:47.314+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:47.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:47.330+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:47.612 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:47.605+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:47.752 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:47.745+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:47.756 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:47.748+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:47.756 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:47.750+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:47.757 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:47.751+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:48.250 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:48.244+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:48.252 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:48.248+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:48.318 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:48.311+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:48.331 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:48.325+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:48.651 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:48.637+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:48.791 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:48.782+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:48.797 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:48.785+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:48.797 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:48.789+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:48.802 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:48.795+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:49.296 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:49.289+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:49.296 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:49.292+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:49.331 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:49.325+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:49.357 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:49.352+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:49.680 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:49.671+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:49.753 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:49.746+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:49.819 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:49.813+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:49.825 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:49.818+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:49.828 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:49.822+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:50.303 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:50.298+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:50.305 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:50.301+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:50.308 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:50.302+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:50.368 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:50.364+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:50.653 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:50.645+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:50.791 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:50.785+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:50.792 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:50.786+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:50.839 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:50.831+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:50.840 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:50.834+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:51.327 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:51.320+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:51.342 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:51.337+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:51.344 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:51.340+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:51.385 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:51.379+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:51.649 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:51.642+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:51.788 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:51.781+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:51.792 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:51.786+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:51.821 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:51.813+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:51.822 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:51.816+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:51.842 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:51.836+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:52.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:52.289+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:52.374 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:52.368+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:52.375 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:52.370+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:52.403 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:52.395+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:52.623 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:52.615+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:52.825 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:52.817+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:52.826 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:52.820+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:52.834 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:52.825+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:52.834 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:52.824+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:52.847 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:52.829+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:53.290 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:53.282+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:53.370 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:53.362+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:53.370 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:53.365+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:53.371 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:53.365+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:53.621 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:53.614+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:53.792 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:53.783+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:53.798 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:53.785+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:53.861 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:53.851+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:53.877 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:53.862+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:53.877 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:53.865+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:54.299 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:54.292+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:54.332 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:54.325+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:54.353 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:54.344+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:54.354 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:54.346+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:54.671 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:54.664+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:54.827 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:54.819+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:54.831 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:54.826+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:54.835 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:54.831+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:54.885 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:54.876+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:54.889 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:54.879+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:55.309 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:55.302+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:55.310 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:55.302+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:55.312 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:55.307+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:55.341 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:55.326+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:55.669 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:55.660+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:55.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:55.786+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:55.868 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:55.859+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:55.869 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:55.863+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:55.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:55.927+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:55.936 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:55.930+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:56.294 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:56.288+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:56.297 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:56.293+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:56.333 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:56.322+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:56.336 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:56.328+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:56.702 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:56.695+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:56.754 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:56.748+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:56.848 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:56.841+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:56.849 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:56.843+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:56.922 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:56.915+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:56.922 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:56.917+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:57.303 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:57.297+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:57.314 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:57.308+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:57.315 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:57.310+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:57.331 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:57.324+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:57.665 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:57.658+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:57.725 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:57.718+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:57.809 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:57.801+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:57.811 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:57.806+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:57.943 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:57.934+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:57.943 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:57.937+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:58.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:58.306+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:58.348 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:58.343+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:58.349 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:58.345+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:58.369 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:58.361+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:58.702 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:58.694+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:58.716 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:58.711+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:58.795 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:58.783+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:58.795 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:58.786+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:58.938 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:58.930+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:58.939 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:58.933+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:06:59.324 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:06:59.318+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:06:59.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:06:59.329+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:06:59.400 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:59.395+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:06:59.402 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:06:59.398+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:06:59.684 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:06:59.678+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:06:59.699 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:06:59.693+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:06:59.752 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:59.744+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:06:59.753 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:06:59.746+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:06:59.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:59.917+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:06:59.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:06:59.920+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:00.305 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:00.301+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:00.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:00.308+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:00.435 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:00.428+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:00.436 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:00.431+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:00.657 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:00.649+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:00.697 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:00.691+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:00.813 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:00.796+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:00.814 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:00.798+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:00.966 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:00.959+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:00.967 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:00.961+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:01.287 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:01.282+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:01.330 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:01.324+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:01.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:01.475+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:01.481 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:01.477+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:01.671 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:01.664+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:01.723 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:01.718+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:01.831 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:01.824+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:01.831 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:01.826+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:01.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:01.984+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:01.993 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:01.987+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:02.261 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:02.255+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:02.322 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:02.311+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:02.495 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:02.490+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:02.496 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:02.491+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:02.659 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:02.654+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:02.742 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:02.736+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:02.885 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:02.877+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:02.886 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:02.880+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:02.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:02.984+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:02.993 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:02.989+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:03.251 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:03.245+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:03.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:03.310+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:03.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:03.465+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:03.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:03.469+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:03.651 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:03.645+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:03.713 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:03.708+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:03.850 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:03.844+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:03.853 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:03.849+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:04.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:04.047+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:04.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:04.049+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:04.222 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:04.215+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:04.351 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:04.344+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:04.446 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:04.440+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:04.447 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:04.441+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:04.701 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:04.695+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:04.723 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:04.717+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:04.832 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:04.821+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:04.845 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:04.838+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:05.076 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:05.071+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:05.077 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:05.072+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:05.260 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:05.255+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:05.311 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:05.306+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:05.494 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:05.489+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:05.495 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:05.491+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:05.673 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:05.667+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:05.710 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:05.704+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:05.848 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:05.840+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:05.850 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:05.842+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:06.104 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:06.096+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:06.104 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:06.099+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:06.255 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:06.249+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:06.272 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:06.267+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:06.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:06.475+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:06.481 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:06.477+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:06.724 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:06.717+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:06.724 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:06.716+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:06.884 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:06.878+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:06.887 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:06.883+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:07.134 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:07.126+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:07.137 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:07.128+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:07.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:07.228+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:07.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:07.307+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:07.481 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:07.476+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:07.482 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:07.477+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:07.702 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:07.695+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:07.711 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:07.705+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:07.916 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:07.909+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:07.917 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:07.912+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:08.172 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:08.165+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:08.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:08.168+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:08.216 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:08.211+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:08.279 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:08.272+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:08.534 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:08.527+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:08.535 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:08.530+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:08.741 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:08.734+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:08.762 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:08.755+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:08.920 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:08.911+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:08.921 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:08.914+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:09.131 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:09.124+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:09.132 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:09.126+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:09.192 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:09.186+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:09.240 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:09.233+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:09.525 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:09.518+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:09.525 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:09.521+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:09.744 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:09.733+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:09.776 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:09.769+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:09.882 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:09.875+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:09.883 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:09.877+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:10.156 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:10.149+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:10.159 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:10.154+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:10.210 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:10.204+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:10.269 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:10.262+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:10.546 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:10.541+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:10.547 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:10.543+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:10.763 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:10.758+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:10.773 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:07:10.767+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e234 v234) 2019-12-11T22:07:10.827 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:10.820+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:10.873 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:10.865+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:10.874 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:10.868+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:11.205 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:11.197+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:11.206 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:11.200+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:11.224 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:11.218+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:11.286 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:11.278+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:11.545 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:11.540+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:11.546 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:11.542+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:11.761 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:11.755+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:11.889 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:11.881+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:11.925 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:11.917+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:11.926 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:11.920+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:12.236 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:12.228+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:12.237 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:12.232+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:12.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:12.231+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:12.244 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:12.236+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:12.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:12.583+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:12.589 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:12.584+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:12.748 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:12.742+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:12.859 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:12.852+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:12.905 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:12.896+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:12.905 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:12.899+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:13.250 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:13.244+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:13.257 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:13.250+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:13.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:13.279+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:13.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:13.282+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:13.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:13.575+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:13.583 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:13.577+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:13.703 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:13.697+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:13.894 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:13.885+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:13.908 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:13.900+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:13.909 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:13.903+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:14.207 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:14.202+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:14.264 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:14.249+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:14.264 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:14.252+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:14.310 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:14.303+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:14.617 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:14.606+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:14.617 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:14.609+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:14.723 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:14.716+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:14.938 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:14.930+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:14.951 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:14.939+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:14.951 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:14.941+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:15.251 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:15.246+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:15.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:15.259+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:15.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:15.262+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:15.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:15.322+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:15.609 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:15.601+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:15.609 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:15.605+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:15.688 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:15.681+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:15.793 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:07:15.789+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e234 v234) 2019-12-11T22:07:15.967 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:15.961+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:16.003 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:15.988+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:16.003 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:15.991+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:16.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:16.271+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:16.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:16.274+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:16.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:16.290+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:16.354 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:16.340+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:16.601 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:16.596+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:16.602 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:16.597+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:16.646 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:16.635+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:16.922 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:16.915+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:17.016 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:17.008+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:17.017 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:17.011+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:17.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:17.287+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:17.300 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:17.292+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:17.330 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:17.322+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:17.347 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:17.341+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:17.568 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:17.558+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:17.569 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:17.559+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:17.610 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:17.603+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:17.919 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:17.911+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:18.066 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:18.058+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:18.066 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:18.061+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:18.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:18.269+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:18.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:18.272+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:18.300 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:18.294+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:18.373 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:18.369+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:18.606 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:18.599+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:18.607 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:18.603+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:18.623 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:18.617+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:18.880 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:18.872+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:19.057 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:19.049+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:19.061 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:19.055+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:19.262 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:19.254+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:19.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:19.313+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:19.322 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:19.316+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:19.412 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:19.405+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:19.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:19.570+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:19.577 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:19.573+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:19.643 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:19.639+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:19.858 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:19.850+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:20.106 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:20.099+0000 1694f700 -1 osd.5 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:06:10.879164+0000 (oldest deadline 2019-12-11T22:06:50.879164+0000) 2019-12-11T22:07:20.109 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:07:20.104+0000 1694f700 -1 osd.5 234 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e232) 2019-12-11T22:07:20.304 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:07:20.296+0000 1694f700 -1 osd.6 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:58.031824+0000 front 2019-12-11T22:05:57.507146+0000 (oldest deadline 2019-12-11T22:06:38.514946+0000) 2019-12-11T22:07:20.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:20.348+0000 1694f700 -1 osd.7 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.200391+0000 front 2019-12-11T22:05:57.261481+0000 (oldest deadline 2019-12-11T22:06:42.489494+0000) 2019-12-11T22:07:20.358 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:07:20.351+0000 1694f700 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:106629 2.20s0 2:04b0bdf7:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (841) in=858b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e227) 2019-12-11T22:07:20.442 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:07:20.436+0000 1694f700 -1 osd.2 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.944230+0000 front 2019-12-11T22:05:53.931637+0000 (oldest deadline 2019-12-11T22:06:38.601355+0000) 2019-12-11T22:07:20.563 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:20.557+0000 1694f700 -1 osd.3 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:57.229336+0000 front 2019-12-11T22:05:57.166838+0000 (oldest deadline 2019-12-11T22:06:38.830647+0000) 2019-12-11T22:07:20.564 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:20.559+0000 1694f700 -1 osd.3 234 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:20.671 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:07:20.665+0000 1694f700 -1 osd.0 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:53.966812+0000 front 2019-12-11T22:05:54.020404+0000 (oldest deadline 2019-12-11T22:06:39.863942+0000) 2019-12-11T22:07:20.868 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:07:20.863+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e234 v234) 2019-12-11T22:07:20.873 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:07:20.867+0000 1694f700 -1 osd.4 234 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:05:55.701841+0000 front 2019-12-11T22:05:49.860252+0000 (oldest deadline 2019-12-11T22:06:40.372313+0000) 2019-12-11T22:07:21.816 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:07:21.809+0000 1694f700 -1 osd.3 235 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107148 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 362299488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e231) 2019-12-11T22:07:22.958 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:07:22.922+0000 20e25700 -1 RGWWatcher::handle_error cookie 362299488 err (107) Transport endpoint is not connected 2019-12-11T22:08:47.767 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:47.759+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:48.726 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:48.720+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:49.739 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:49.733+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:50.767 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:50.760+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:51.764 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:51.758+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:52.816 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:52.810+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:53.838 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:53.833+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:54.837 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:54.831+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:08:54.838 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:54.832+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:55.312 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:08:55.307+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:08:55.408 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:08:55.403+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:08:55.537 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:08:55.531+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:08:55.887 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:55.882+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:08:55.890 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:55.886+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:56.060 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:08:56.053+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:08:56.293 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:08:56.282+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:08:56.382 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:08:56.376+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:08:56.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:08:56.540+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:08:56.932 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:56.927+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:08:56.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:56.933+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:57.038 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:08:57.032+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:08:57.039 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:08:57.032+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:08:57.302 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:08:57.296+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:08:57.379 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:08:57.372+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:08:57.591 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:08:57.584+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:08:57.984 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:57.978+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:08:57.987 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:57.982+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:58.018 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:08:58.010+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:08:58.080 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:08:58.070+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:08:58.213 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:08:58.205+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:08:58.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:08:58.339+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:08:58.424 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:08:58.417+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:08:58.562 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:08:58.556+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:08:59.009 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:59.004+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:08:59.009 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:08:59.005+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:08:59.034 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:08:59.028+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:08:59.063 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:08:59.058+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:08:59.257 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:08:59.250+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:08:59.351 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:08:59.338+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:08:59.428 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:08:59.422+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:08:59.602 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:08:59.596+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:00.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:00.000+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:00.007 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:00.003+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:00.036 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:00.029+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:00.072 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:00.066+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:00.249 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:00.244+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:00.340 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:00.335+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:00.389 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:00.384+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:00.606 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:00.600+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:00.967 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:00.962+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:00.968 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:00.963+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:00.992 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:00.986+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:01.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:01.086+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:01.243 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:01.237+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:01.364 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:01.359+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:01.422 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:01.418+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:01.653 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:01.647+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:01.924 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:01.918+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:01.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:01.921+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:01.985 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:01.981+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:02.076 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:02.071+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:02.197 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:02.191+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:02.380 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:02.374+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:02.388 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:02.381+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:02.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:02.679+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:02.901 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:02.894+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:02.903 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:02.898+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:02.962 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:02.957+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:03.068 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:03.063+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:03.236 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:03.230+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:03.411 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:03.406+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:03.416 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:03.410+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:03.693 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:03.686+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:03.910 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:03.904+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:03.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:03.907+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:03.997 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:03.991+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:04.100 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:04.095+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:04.209 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:04.204+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:04.378 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:04.371+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:04.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:04.380+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:04.679 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:04.672+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:04.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:04.879+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:04.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:04.880+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:05.039 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:05.034+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:05.123 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:05.118+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:05.176 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:05.171+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:05.389 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:05.383+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:05.413 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:05.407+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:05.650 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:05.643+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:05.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:05.848+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:05.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:05.850+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:06.049 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:06.044+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:06.108 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:06.103+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:06.133 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:06.128+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:06.369 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:06.364+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:06.393 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:06.386+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:06.630 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:06.624+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:06.900 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:06.894+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:06.904 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:06.898+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:07.117 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:07.095+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:07.117 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:07.104+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:07.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:07.109+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:07.413 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:07.407+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:07.450 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:07.423+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:07.677 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:07.670+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:07.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:07.933+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:07.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:07.935+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:08.091 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:08.086+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:08.129 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:08.123+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:08.153 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:08.140+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:08.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:08.454+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:08.483 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:08.477+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:08.712 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:08.705+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:08.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:08.924+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:08.932 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:08.926+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:09.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:09.054+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:09.120 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:09.114+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:09.205 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:09.193+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:09.423 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:09.418+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:09.493 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:09.486+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:09.687 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:09.680+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:09.959 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:09.952+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:09.962 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:09.956+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:10.089 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:10.078+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:10.091 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:10.085+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:10.179 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:10.173+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:10.400 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:10.395+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:10.525 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:10.519+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:10.670 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:10.664+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:10.922 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:10.917+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:10.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:10.919+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:11.062 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:11.055+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:11.062 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:11.057+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:11.134 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:11.128+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:11.442 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:11.438+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:11.501 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:11.495+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:11.679 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:11.673+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:11.952 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:11.943+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:11.953 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:11.944+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:12.022 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:12.017+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:12.037 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:12.032+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:12.120 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:12.114+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:12.412 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:12.406+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:12.543 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:12.535+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:12.728 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:12.721+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:13.001 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:12.995+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:13.002 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:12.996+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:13.061 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:13.043+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:13.061 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:13.048+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:13.076 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:13.070+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:13.403 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:13.398+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:13.591 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:13.584+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:13.773 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:13.766+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:14.028 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:14.023+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:14.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:14.046+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:14.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:14.048+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:14.096 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:14.091+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:14.102 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:14.097+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:14.445 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:14.441+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:14.560 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:14.554+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:14.816 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:14.810+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:15.071 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:15.064+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:15.087 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:15.081+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:15.087 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:15.082+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:15.093 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:15.089+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:15.120 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:15.115+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:15.419 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:15.414+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:15.574 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:15.567+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:15.802 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:15.794+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:16.083 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:16.070+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:16.084 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:16.057+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:16.084 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:16.058+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:16.113 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:16.108+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:16.114 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:16.107+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:16.405 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:16.399+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:16.597 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:16.589+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:16.842 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:16.835+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:17.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:17.037+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:17.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:17.038+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:17.046 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:17.042+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:17.086 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:17.080+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:17.121 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:17.117+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:17.456 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:17.451+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:17.555 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:17.549+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:17.804 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:17.797+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:18.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:18.011+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:18.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:18.013+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:18.027 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:18.022+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:18.074 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:18.069+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:18.127 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:18.121+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:18.488 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:18.483+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:18.556 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:18.550+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:18.845 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:18.838+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:19.006 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:18.997+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:19.009 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:19.003+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:19.078 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:19.073+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:19.080 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:19.074+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:19.156 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:19.151+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:19.477 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:19.472+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:19.544 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:19.537+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:19.884 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:19.878+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:20.008 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:20.000+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:20.009 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:20.002+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:20.119 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:20.114+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:20.133 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:20.122+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:20.141 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:20.136+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:20.477 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:20.465+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:20.544 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:20.536+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:20.881 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:20.874+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:20.975 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:20.969+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:20.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:20.973+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:21.137 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:21.131+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:21.148 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:21.143+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:21.162 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:21.156+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:21.438 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:21.433+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:21.571 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:21.564+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:21.845 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:21.837+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:22.013 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:22.007+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:22.015 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:22.010+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:22.176 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:22.167+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:22.178 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:22.172+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:22.185 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:22.179+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:22.469 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:22.464+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:22.530 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:22.524+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:22.819 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:22.813+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:23.064 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:23.057+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:23.064 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:23.058+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:23.192 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:23.187+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:23.205 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:23.200+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:23.209 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:23.204+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:23.441 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:23.436+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:23.575 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:23.569+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:23.777 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:23.771+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:24.090 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:24.082+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:24.094 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:24.083+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:24.193 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:24.188+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:24.200 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:24.195+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:24.232 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:24.228+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:24.435 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:24.429+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:24.573 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:24.566+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:24.822 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:24.808+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:25.089 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:25.083+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:25.092 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:25.087+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:25.200 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:25.194+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:25.205 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:25.200+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:25.222 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:25.217+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:25.396 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:25.391+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:25.553 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:25.546+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:25.846 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:25.839+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:26.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:26.051+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:26.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:26.055+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:26.179 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:26.173+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:26.190 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:26.184+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:26.253 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:26.248+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:26.404 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:26.398+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:26.557 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:26.551+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:26.877 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:26.870+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:27.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:27.080+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:27.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:27.081+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:27.166 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:27.160+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:27.225 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:27.220+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:27.258 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:27.252+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:27.428 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:27.423+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:27.527 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:27.520+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:27.837 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:27.829+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:28.069 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:28.063+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:28.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:28.066+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:28.199 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:28.194+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:28.246 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:28.241+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:28.268 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:28.262+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:28.391 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:28.386+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:28.556 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:28.549+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:28.806 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:28.799+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:29.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:29.112+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:29.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:29.113+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:29.175 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:29.170+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:29.217 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:29.212+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:29.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:29.292+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:29.435 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:29.430+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:29.566 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:29.559+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:29.802 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:29.795+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:30.090 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:30.083+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:30.091 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:30.084+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:30.180 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:30.175+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:30.259 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:30.254+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:30.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:30.276+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:30.415 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:30.410+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:30.595 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:30.589+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:30.804 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:09:30.798+0000 1694f700 -1 osd.4 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.930958+0000 front 2019-12-11T22:08:13.037251+0000 (oldest deadline 2019-12-11T22:08:54.591807+0000) 2019-12-11T22:09:31.121 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:31.108+0000 1694f700 -1 osd.7 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.669399+0000 front 2019-12-11T22:08:11.668150+0000 (oldest deadline 2019-12-11T22:08:54.619250+0000) 2019-12-11T22:09:31.121 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:31.110+0000 1694f700 -1 osd.7 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:31.203 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:09:31.197+0000 1694f700 -1 osd.0 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.850968+0000 front 2019-12-11T22:08:12.441858+0000 (oldest deadline 2019-12-11T22:08:56.917442+0000) 2019-12-11T22:09:31.278 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:09:31.271+0000 1694f700 -1 osd.2 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:11.347445+0000 front 2019-12-11T22:08:10.703648+0000 (oldest deadline 2019-12-11T22:08:55.345630+0000) 2019-12-11T22:09:31.313 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:09:31.309+0000 1694f700 -1 osd.3 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:12.700081+0000 front 2019-12-11T22:08:12.702695+0000 (oldest deadline 2019-12-11T22:08:57.338069+0000) 2019-12-11T22:09:31.373 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:09:31.368+0000 1694f700 -1 osd.1 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.603143+0000 front 2019-12-11T22:08:09.729663+0000 (oldest deadline 2019-12-11T22:08:54.805438+0000) 2019-12-11T22:09:31.539 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:09:31.531+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 47sec e239 v239) 2019-12-11T22:09:31.571 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:09:31.564+0000 1694f700 -1 osd.6 239 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:08:10.240441+0000 front 2019-12-11T22:08:10.931652+0000 (oldest deadline 2019-12-11T22:08:54.319279+0000) 2019-12-11T22:09:32.541 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:32.474+0000 1694f700 -1 osd.7 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:33.730 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:09:33.705+0000 1694f700 -1 osd.7 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107370 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~0,setxattr user.rgw.manifest (1352) in=1369b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e239) 2019-12-11T22:09:36.687 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:37:23.616 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:36.699 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:37:23.629 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:36.704 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:37:23.634 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:36.708 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:37:23.638 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:36.713 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:37:23.615 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:36.723 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:37:23.623 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:36.947 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:37:23.878 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:36.977 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:37:23.878 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:09:37.045 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:37:23.976 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:10:01.955 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:01.949+0000 1694f700 -1 osd.4 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107484 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 57] snapc 0=[] ondisk+write+known_if_redirected e240) 2019-12-11T22:10:07.135 INFO:teuthology.orchestra.run.smithi079.stderr:testNameLimit (test.functional.tests.TestFile) ... ok 2019-12-11T22:10:07.173 INFO:teuthology.orchestra.run.smithi079.stderr:testNoContentLengthForPut (test.functional.tests.TestFile) ... ok 2019-12-11T22:10:07.555 INFO:teuthology.orchestra.run.smithi079.stderr:testPostOnFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-11T22:10:09.828 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:09.710+0000 35f8f700 -1 osd.0 243 build_incremental_map_msg missing incremental map 243 2019-12-11T22:10:09.852 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:09.734+0000 35f8f700 -1 osd.0 243 build_incremental_map_msg missing incremental map 243 2019-12-11T22:10:09.919 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:09.817+0000 35f8f700 -1 osd.0 243 build_incremental_map_msg missing incremental map 243 2019-12-11T22:10:10.083 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:09.870+0000 31786700 -1 osd.0 243 build_incremental_map_msg missing incremental map 243 2019-12-11T22:10:10.232 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:10.159+0000 3478c700 -1 osd.0 243 build_incremental_map_msg missing incremental map 243 2019-12-11T22:10:20.739 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:20.734+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:21.790 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:21.786+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:22.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:22.827+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:23.801 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:23.795+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:24.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:24.781+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:25.760 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:25.755+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:26.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:26.756+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:27.727 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:27.722+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:28.406 INFO:teuthology.orchestra.run.smithi079.stderr:testRangedGets (test.functional.tests.TestFile) ... ok 2019-12-11T22:10:28.725 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:28.720+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:29.727 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:29.723+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:30.164 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:30.156+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:30.189 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:30.182+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:30.723 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:30.712+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:31.157 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:31.152+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:31.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:31.203+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:31.697 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:31.691+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:31.698 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:31.693+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:32.118 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:32.113+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:32.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:32.173+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:32.269 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:32.262+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:32.706 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:32.701+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:32.706 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:32.702+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:32.938 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:32.930+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:33.128 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:33.121+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:33.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:33.185+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:33.319 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:33.313+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:33.670 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:33.666+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:33.672 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:33.668+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:33.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:33.926+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:34.095 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:34.089+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:34.229 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:34.221+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:34.274 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:34.268+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:34.706 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:34.701+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:34.709 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:34.705+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:34.899 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:34.889+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:35.063 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:35.057+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:35.230 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:35.223+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:35.261 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:35.257+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:35.370 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:35.363+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:35.715 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:35.708+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:35.715 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:35.711+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:35.892 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:35.884+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:36.087 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:36.082+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:36.213 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:36.206+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:36.279 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:36.273+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:36.346 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:36.338+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:36.702 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:36.697+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:36.703 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:36.699+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:36.861 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:36.854+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:37.103 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:37.097+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:37.193 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:37.185+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:37.244 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:37.237+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:37.398 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:37.391+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:37.718 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:37.712+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:37.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:37.715+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:37.896 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:37.888+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:38.091 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:38.085+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:38.158 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:38.151+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:38.245 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:38.239+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:38.423 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:38.416+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:38.682 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:38.676+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:38.682 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:38.678+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:38.893 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:38.885+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:39.118 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:39.112+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:39.139 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:39.131+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:39.272 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:39.268+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:39.460 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:39.452+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:39.696 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:39.691+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:39.700 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:39.696+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:39.919 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:39.910+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:40.113 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:40.107+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:40.122 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:40.115+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:40.283 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:40.278+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:40.484 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:40.477+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:40.715 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:40.709+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:40.715 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:40.711+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:40.811 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:40.785+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:40.906 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:40.899+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:41.088 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:41.079+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:41.140 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:41.133+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:41.298 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:41.292+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:41.508 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:41.500+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:41.728 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:41.723+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:41.730 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:41.725+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:41.764 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:41.757+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:41.904 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:41.897+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:42.086 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:42.079+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:42.146 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:42.139+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:42.299 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:42.294+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:42.514 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:42.508+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:42.751 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:42.746+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:42.751 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:42.747+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:42.799 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:42.792+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:42.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:42.918+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:43.114 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:43.109+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:43.181 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:43.175+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:43.341 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:43.336+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:43.512 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:43.503+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:43.747 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:43.742+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:43.748 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:43.744+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:43.803 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:43.798+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:43.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:43.918+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:44.096 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:44.092+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:44.174 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:44.169+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:44.315 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:44.309+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:44.476 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:44.468+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:44.717 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:44.709+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:44.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:44.713+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:44.814 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:44.807+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:44.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:44.876+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:45.062 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:45.057+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:45.218 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:45.210+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:45.273 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:45.267+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:45.503 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:45.496+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:45.728 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:45.722+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:45.730 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:45.724+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:45.858 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:45.850+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:45.874 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:45.867+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:46.111 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:46.106+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:46.213 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:46.206+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:46.286 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:46.281+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:46.519 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:46.512+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:46.779 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:46.774+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:46.780 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:46.775+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:46.855 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:46.849+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:46.861 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:46.855+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:47.110 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:47.104+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:47.265 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:47.258+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:47.271 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:47.266+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:47.482 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:47.471+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:47.812 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:47.805+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:47.813 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:47.808+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:47.824 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:47.818+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:47.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:47.823+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:47.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:47.903+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:48.113 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:48.108+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:48.251 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:48.246+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:48.258 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:48.252+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:48.512 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:48.505+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:48.798 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:48.792+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:48.798 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:48.794+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:48.810 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:48.802+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:48.812 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:48.806+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:48.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:48.875+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:49.085 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:49.079+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:49.257 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:49.250+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:49.298 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:49.292+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:49.542 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:49.536+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:49.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:49.777+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:49.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:49.780+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:49.806 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:49.798+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:49.807 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:49.801+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:49.913 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:49.906+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:50.041 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:50.036+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:50.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:50.238+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:50.271 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:50.266+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:50.549 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:50.543+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:50.766 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:50.757+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:50.766 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:50.760+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:50.858 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:50.850+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:50.858 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:50.853+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:50.885 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:50.876+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:51.094 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:51.070+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:51.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:51.291+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:51.325 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:51.319+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:51.581 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:51.574+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:51.755 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:51.748+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:51.757 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:51.751+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:51.823 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:51.817+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:51.826 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:51.822+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:51.908 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:51.897+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:52.114 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:52.108+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:52.308 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:52.302+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:52.331 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:52.324+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:52.624 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:52.619+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:52.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:52.713+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:52.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:52.715+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:52.783 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:52.777+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:52.787 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:52.782+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:52.898 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:52.890+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:53.121 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:53.115+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:53.289 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:53.284+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:53.359 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:53.352+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:53.613 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:53.607+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:53.675 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:53.669+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:53.678 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:53.674+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:53.799 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:53.791+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:53.800 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:53.794+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:53.853 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:53.847+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:54.114 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:54.106+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:54.329 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:54.324+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:54.331 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:54.323+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:54.586 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:54.580+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:54.667 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:54.659+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:54.671 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:54.665+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:54.837 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:54.830+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:54.850 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:54.844+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:54.853 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:54.846+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:55.084 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:55.079+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:55.289 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:55.281+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:55.378 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:55.372+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:55.613 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:55.605+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:55.651 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:55.643+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:55.652 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:55.647+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:55.873 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:55.866+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:55.877 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:55.869+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:55.878 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:55.872+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:56.073 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:56.068+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:56.301 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:56.293+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:56.410 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:56.404+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:56.620 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:56.612+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:56.672 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:56.666+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:56.674 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:56.670+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:56.890 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:56.883+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:56.893 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:56.887+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:56.896 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:56.891+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:57.119 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:57.111+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:57.284 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:57.277+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:57.446 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:57.440+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:57.657 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:57.650+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:57.723 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:57.717+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:57.724 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:57.719+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:57.894 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:57.885+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:57.895 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:57.888+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:57.914 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:57.906+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:58.140 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:58.134+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:58.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:58.309+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:58.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:58.402+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:58.710 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:58.702+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:58.718 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:58.712+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:58.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:58.715+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:58.929 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:58.921+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:58.930 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:58.924+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:10:58.951 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:58.943+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:10:59.189 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:10:59.183+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:10:59.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:10:59.308+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:10:59.366 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:10:59.360+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:10:59.731 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:10:59.721+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:10:59.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:59.760+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:10:59.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:10:59.761+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:10:59.928 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:59.921+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:10:59.931 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:10:59.926+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:00.000 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:10:59.993+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:00.237 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:00.232+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:00.309 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:00.302+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:00.349 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:00.343+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:00.716 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:00.708+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:00.789 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:00.783+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:00.789 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:00.785+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:00.964 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:00.954+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:00.965 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:00.957+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:01.030 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:01.022+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:01.266 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:01.261+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:01.333 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:01.326+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:01.372 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:01.367+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:01.728 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:01.721+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:01.797 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:11:01.789+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e244 v244) 2019-12-11T22:11:01.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:01.835+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:01.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:01.837+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:01.952 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:01.938+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:01.959 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:01.941+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:01.999 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:01.993+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:02.234 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:02.229+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:02.300 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:02.293+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:02.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:02.415+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:02.720 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:02.714+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:02.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:02.872+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:02.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:02.874+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:02.904 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:02.896+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:02.905 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:02.898+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:03.028 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:03.021+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:03.250 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:03.244+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:03.348 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:03.341+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:03.387 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:03.381+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:03.734 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:03.727+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:03.892 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:03.885+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:03.894 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:03.889+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:03.895 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:03.890+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:03.897 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:03.893+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:04.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:04.052+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:04.289 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:04.278+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:04.401 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:04.394+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:04.438 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:04.432+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:04.787 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:04.778+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:04.911 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:04.897+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:04.912 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:04.900+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:04.923 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:04.913+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:04.924 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:04.917+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:05.069 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:05.061+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:05.255 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:05.246+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:05.439 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:05.431+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:05.455 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:05.448+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:05.784 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:05.776+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:05.885 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:05.878+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:05.886 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:05.881+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:05.902 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:05.892+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:05.903 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:05.895+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:06.125 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:06.099+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:06.274 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:06.268+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:06.444 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:06.436+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:06.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:06.467+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:06.779 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:06.770+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:06.813 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:11:06.805+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e244 v244) 2019-12-11T22:11:06.897 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:06.890+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:06.898 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:06.892+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:06.925 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:06.918+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:06.929 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:06.923+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:07.160 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:07.152+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:07.309 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:07.302+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:07.438 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:07.431+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:07.522 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:07.515+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:07.813 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:07.805+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:07.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:07.871+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:07.879 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:07.873+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:07.980 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:07.971+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:07.981 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:07.974+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:08.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:08.107+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:08.306 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:08.300+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:08.458 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:08.451+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:08.557 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:08.551+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:08.800 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:08.792+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:08.934 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:08.914+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:08.934 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:08.917+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:08.970 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:08.961+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:08.970 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:08.964+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:09.084 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:09.076+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:09.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:09.328+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:09.488 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:09.480+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:09.589 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:09.584+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:09.756 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:09.748+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:09.943 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:09.934+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:09.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:09.938+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:09.982 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:09.974+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:09.983 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:09.977+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:10.073 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:10.064+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:10.297 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:10.291+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:10.476 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:10.468+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:10.549 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:10.543+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:10.784 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:10.777+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:10.906 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:10.900+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:10.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:10.905+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:11.009 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:11.001+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:11.010 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:11.004+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:11.105 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:11.098+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:11.349 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:11.342+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:11.447 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:11.440+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:11.595 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:11.587+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:11.746 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:11.740+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:11.828 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:11:11.823+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e244 v244) 2019-12-11T22:11:11.950 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:11.945+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:11.953 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:11.948+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:12.018 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:12.011+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:12.019 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:12.013+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:12.156 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:12.148+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:12.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:12.325+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:12.490 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:12.483+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:12.612 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:12.607+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:12.727 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:12.720+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:12.975 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:12.964+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:12.975 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:12.967+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:13.033 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:13.025+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:13.034 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:13.028+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:13.199 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:13.192+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:13.344 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:13.338+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:13.479 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:13.462+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:13.579 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:13.574+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:13.777 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:13.768+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:13.942 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:13.937+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:13.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:13.941+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:14.002 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:13.994+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:14.005 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:13.999+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:14.179 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:14.172+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:14.363 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:14.358+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:14.455 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:14.444+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:14.545 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:14.538+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:14.771 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:14.761+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:14.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:14.924+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:14.933 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:14.928+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:14.982 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:14.975+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:14.986 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:14.980+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:15.189 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:15.169+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:15.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:15.171+0000 1694f700 -1 osd.7 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:15.411 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:15.407+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:15.448 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:15.442+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:15.576 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:15.570+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:15.813 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:15.805+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:15.960 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:15.950+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:15.960 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:15.954+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:15.972 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:15.965+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:15.974 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:15.968+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:16.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:16.131+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:16.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:16.134+0000 1694f700 -1 osd.7 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:16.424 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:16.416+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:16.458 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:16.451+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:16.536 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:16.529+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:16.830 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:16.823+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:16.849 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:11:16.841+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e244 v244) 2019-12-11T22:11:16.934 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:16.928+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:16.934 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:16.930+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:16.967 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:16.957+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:16.968 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:16.960+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:17.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:17.141+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:17.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:17.143+0000 1694f700 -1 osd.7 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:17.462 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:17.443+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:17.494 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:17.489+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:17.509 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:17.505+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:17.849 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:17.842+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:17.917 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:17.910+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:17.918 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:17.913+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:17.988 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:17.982+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:18.002 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:17.987+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:18.162 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:18.155+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:18.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:18.157+0000 1694f700 -1 osd.7 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:18.457 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:18.450+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:18.510 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:18.504+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:18.523 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:18.518+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:18.877 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:18.870+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:18.941 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:18.929+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:18.943 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:18.934+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:19.044 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:19.036+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:19.045 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:19.039+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:19.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:19.112+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:19.122 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:19.117+0000 1694f700 -1 osd.7 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:19.492 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:19.487+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:19.507 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:19.500+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:19.512 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:19.507+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:19.866 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:19.854+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:19.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:19.939+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:19.947 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:19.941+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:20.054 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:20.046+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:20.055 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:20.049+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:20.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:20.157+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:20.166 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:20.160+0000 1694f700 -1 osd.7 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:20.476 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:20.471+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:20.494 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:20.488+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:20.528 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:20.521+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:20.834 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:20.826+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:20.962 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:20.954+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:20.963 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:20.958+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:21.041 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:21.033+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:10:07.240263+0000 (oldest deadline 2019-12-11T22:10:47.240263+0000) 2019-12-11T22:11:21.042 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:21.036+0000 1694f700 -1 osd.5 244 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:21.210 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:21.202+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.600258+0000 front 2019-12-11T22:09:48.554197+0000 (oldest deadline 2019-12-11T22:10:32.515803+0000) 2019-12-11T22:11:21.211 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:21.205+0000 1694f700 -1 osd.7 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:21.491 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:11:21.484+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.364276+0000 front 2019-12-11T22:09:48.715997+0000 (oldest deadline 2019-12-11T22:10:31.857027+0000) 2019-12-11T22:11:21.493 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:11:21.485+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.168898+0000 front 2019-12-11T22:09:47.195726+0000 (oldest deadline 2019-12-11T22:10:29.231258+0000) 2019-12-11T22:11:21.494 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:11:21.488+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:46.087682+0000 front 2019-12-11T22:09:46.082655+0000 (oldest deadline 2019-12-11T22:10:29.551687+0000) 2019-12-11T22:11:21.877 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:11:21.868+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:48.905021+0000 front 2019-12-11T22:09:48.724394+0000 (oldest deadline 2019-12-11T22:10:34.606473+0000) 2019-12-11T22:11:21.899 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:11:21.889+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e244 v244) 2019-12-11T22:11:21.970 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:21.963+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:09:47.919172+0000 front 2019-12-11T22:09:47.941917+0000 (oldest deadline 2019-12-11T22:10:31.377873+0000) 2019-12-11T22:11:21.971 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:21.966+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:22.339 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:11:22.220+0000 1694f700 -1 osd.5 245 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e243) 2019-12-11T22:11:22.447 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:22.414+0000 1694f700 -1 osd.7 245 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:23.018 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:11:23.003+0000 1694f700 -1 osd.3 245 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:107519 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5419397328 gen 1] snapc 0=[] ondisk+write+known_if_redirected e241) 2019-12-11T22:11:23.740 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:11:23.701+0000 1694f700 -1 osd.7 246 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:107730 2.0s0 2:0352f967:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.670_1e42e9f2081e47a1b6b7cafd7a6ed799:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~1014 in=1014b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (33) in=54b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e244) 2019-12-11T22:11:24.260 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:11:24.252+0000 20e25700 -1 RGWWatcher::handle_error cookie 5419397328 err (107) Transport endpoint is not connected 2019-12-11T22:12:18.491 INFO:teuthology.orchestra.run.smithi079.stderr:testSerialization (test.functional.tests.TestFile) ... ok 2019-12-11T22:12:44.339 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:44.334+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:44.441 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:44.436+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:44.453 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:44.448+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:45.338 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:45.334+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:45.456 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:45.452+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:45.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:45.466+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:45.505 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:45.500+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:45.557 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:45.552+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:46.354 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:46.348+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:46.454 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:46.448+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:46.482 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:46.477+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:46.507 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:46.496+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:46.570 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:46.565+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:47.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:47.204+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:47.347 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:47.342+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:47.491 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:47.487+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:47.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:47.497+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:47.537 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:47.532+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:47.543 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:47.538+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:48.170 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:48.165+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:48.345 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:48.338+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:48.360 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:48.356+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:48.493 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:48.488+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:48.523 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:48.517+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:48.528 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:48.521+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:48.542 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:48.536+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:49.136 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:49.131+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:49.359 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:49.355+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:49.364 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:49.360+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:49.512 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:49.508+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:49.519 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:49.515+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:49.534 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:49.527+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:49.548 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:49.541+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:50.145 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:50.139+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:50.389 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:50.384+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:50.413 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:50.409+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:50.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:50.495+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:50.501 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:50.495+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:50.530 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:50.525+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:50.568 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:50.561+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:51.187 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:51.182+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:51.392 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:51.386+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:51.419 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:51.413+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:51.492 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:51.488+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:51.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:51.495+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:51.522 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:51.517+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:51.619 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:51.612+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:52.215 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:52.207+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:52.425 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:52.421+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:52.454 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:52.449+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:52.498 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:52.491+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:52.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:52.501+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:52.525 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:52.520+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:52.673 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:52.666+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:53.191 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:53.184+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:53.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:53.415+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:53.428 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:53.424+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:53.499 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:53.492+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:53.541 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:53.536+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:53.570 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:53.565+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:53.639 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:53.632+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:54.207 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:54.201+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:54.415 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:54.409+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:54.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:54.449+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:54.494 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:54.488+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:54.547 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:54.542+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:54.563 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:54.558+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:54.617 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:54.611+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:55.211 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:55.205+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:55.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:55.382+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:55.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:55.466+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:55.481 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:55.477+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:55.572 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:55.568+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:55.600 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:55.595+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:55.658 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:55.652+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:56.219 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:56.212+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:56.403 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:56.398+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:56.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:56.429+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:56.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:56.450+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:56.571 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:56.566+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:56.583 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:56.578+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:56.649 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:56.643+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:57.215 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:57.209+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:57.385 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:57.380+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:57.413 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:57.406+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:57.413 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:57.409+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:57.563 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:57.559+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:57.564 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:57.560+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:57.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:57.690+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:58.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:58.169+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:58.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:58.402+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:58.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:58.436+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:58.445 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:58.436+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:58.556 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:58.551+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:58.569 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:58.564+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:58.705 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:58.699+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:12:59.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:12:59.163+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:12:59.368 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:12:59.363+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:12:59.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:12:59.428+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:12:59.488 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:12:59.481+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:12:59.542 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:12:59.536+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:12:59.582 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:12:59.577+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:12:59.690 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:12:59.684+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:00.213 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:00.205+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:00.332 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:00.327+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:00.487 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:00.483+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:00.497 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:00.493+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:00.503 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:00.493+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:00.549 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:00.545+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:00.673 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:00.667+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:01.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:01.242+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:01.289 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:01.284+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:01.495 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:01.488+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:01.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:01.512+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:01.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:01.533+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:01.575 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:01.570+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:01.676 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:01.670+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:02.295 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:02.288+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:02.306 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:02.302+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:02.524 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:02.517+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:02.543 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:02.534+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:02.564 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:02.557+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:02.600 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:02.595+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:02.717 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:02.710+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:03.266 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:03.260+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:03.291 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:03.285+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:03.493 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:03.487+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:03.578 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:03.571+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:03.621 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:03.613+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:03.648 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:03.642+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:03.714 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:03.707+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:04.238 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:04.234+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:04.274 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:04.268+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:04.481 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:04.468+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:04.582 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:04.577+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:04.616 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:04.611+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:04.626 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:04.620+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:04.749 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:04.741+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:05.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:05.259+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:05.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:05.276+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:05.458 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:05.452+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:05.600 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:05.595+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:05.614 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:05.610+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:05.644 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:05.640+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:05.733 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:05.727+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:06.255 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:06.240+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:06.288 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:06.278+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:06.485 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:06.477+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:06.608 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:06.604+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:06.615 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:06.610+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:06.655 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:06.650+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:06.780 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:06.774+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:07.256 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:07.252+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:07.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:07.289+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:07.477 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:07.471+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:07.611 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:07.606+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:07.628 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:07.623+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:07.669 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:07.664+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:07.825 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:07.819+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:08.272 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:08.265+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:08.278 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:08.274+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:08.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:08.494+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:08.587 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:08.582+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:08.643 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:08.639+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:08.660 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:08.655+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:08.863 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:08.857+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:09.256 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:09.249+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:09.286 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:09.280+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:09.502 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:09.495+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:09.552 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:09.547+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:09.647 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:09.635+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:09.670 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:09.665+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:09.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:09.899+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:10.222 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:10.216+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:10.279 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:10.271+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:10.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:10.495+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:10.567 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:10.562+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:10.594 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:10.590+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:10.717 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:10.712+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:10.869 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:10.862+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:11.211 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:11.204+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:11.273 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:11.266+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:11.482 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:11.476+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:11.542 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:11.537+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:11.628 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:11.623+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:11.743 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:11.737+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:11.874 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:11.861+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:12.181 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:12.166+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:12.275 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:12.265+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:12.482 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:12.476+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:12.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:12.496+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:12.615 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:12.610+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:12.724 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:12.719+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:12.880 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:12.873+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:13.189 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:13.183+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:13.226 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:13.220+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:13.487 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:13.482+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:13.508 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:13.502+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:13.584 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:13.580+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:13.717 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:13.710+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:13.891 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:13.884+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:14.218 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:14.213+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:14.218 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:14.212+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:14.472 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:14.465+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:14.488 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:14.483+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:14.539 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:14.533+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:14.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:14.728+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:14.846 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:14.839+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:15.210 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:15.205+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:15.227 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:15.220+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:15.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:15.438+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:15.448 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:15.443+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:15.559 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:15.555+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:15.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:15.727+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:15.834 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:15.827+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:16.193 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:16.186+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:16.253 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:16.248+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:16.409 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:16.404+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:16.487 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:16.471+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:16.601 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:16.597+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:16.689 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:16.684+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:16.836 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:16.829+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:17.213 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:17.197+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:17.232 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:13:17.220+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e249 v249) 2019-12-11T22:13:17.264 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:17.254+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:17.385 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:17.376+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:17.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:17.499+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:17.572 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:17.564+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:17.718 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:17.712+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:17.808 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:17.801+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:18.206 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:18.200+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:18.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:18.288+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:18.373 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:18.367+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:18.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:18.514+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:18.567 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:18.559+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:18.679 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:18.673+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:18.852 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:18.845+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:19.184 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:19.178+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:19.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:19.288+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:19.362 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:19.356+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:19.543 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:19.537+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:19.578 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:19.571+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:19.672 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:19.665+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:19.860 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:19.853+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:20.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:20.167+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:20.303 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:20.297+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:20.364 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:20.359+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:20.567 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:20.561+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:20.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:20.565+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:20.650 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:20.645+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:20.864 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:20.858+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:21.201 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:21.184+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:21.302 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:21.297+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:21.393 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:13:21.386+0000 1694f700 -1 osd.3 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:03.871792+0000 front 2019-12-11T22:11:59.181192+0000 (oldest deadline 2019-12-11T22:12:43.773543+0000) 2019-12-11T22:13:21.529 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:13:21.524+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.919067+0000 front 2019-12-11T22:12:02.905670+0000 (oldest deadline 2019-12-11T22:12:44.897679+0000) 2019-12-11T22:13:21.585 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:13:21.578+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.330848+0000 front 2019-12-11T22:12:01.335593+0000 (oldest deadline 2019-12-11T22:12:43.993941+0000) 2019-12-11T22:13:21.655 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:13:21.649+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.776538+0000 front 2019-12-11T22:12:03.055747+0000 (oldest deadline 2019-12-11T22:12:47.882317+0000) 2019-12-11T22:13:21.854 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:13:21.847+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.223977+0000 front 2019-12-11T22:12:01.711847+0000 (oldest deadline 2019-12-11T22:12:44.956910+0000) 2019-12-11T22:13:22.211 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:13:22.204+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:02.121961+0000 front 2019-12-11T22:12:02.125195+0000 (oldest deadline 2019-12-11T22:12:46.803429+0000) 2019-12-11T22:13:22.296 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:13:22.288+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e249 v249) 2019-12-11T22:13:22.349 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:13:22.343+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:12:01.185043+0000 front 2019-12-11T22:12:01.300934+0000 (oldest deadline 2019-12-11T22:12:44.237884+0000) 2019-12-11T22:13:26.889 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:41:13.820 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:26.895 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:41:13.826 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:26.905 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:41:13.806 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:26.906 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:41:13.836 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:26.910 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:41:13.841 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:26.913 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:41:13.815 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:27.193 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:41:14.115 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:27.221 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:41:14.150 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:13:27.221 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:41:14.117 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:01.347 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:01.339+0000 11caf700 -1 osd.5 255 build_incremental_map_msg missing incremental map 255 2019-12-11T22:14:01.396 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:01.369+0000 30f85700 -1 osd.5 255 build_incremental_map_msg missing incremental map 255 2019-12-11T22:14:01.413 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:01.390+0000 31786700 -1 osd.5 255 build_incremental_map_msg missing incremental map 255 2019-12-11T22:14:01.676 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:01.632+0000 30f85700 -1 osd.5 255 build_incremental_map_msg missing incremental map 255 2019-12-11T22:14:01.801 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:01.759+0000 31f87700 -1 osd.5 255 build_incremental_map_msg missing incremental map 255 2019-12-11T22:14:02.723 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:02.713+0000 34f8d700 -1 osd.5 255 build_incremental_map_msg missing incremental map 255 2019-12-11T22:14:02.758 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:02.752+0000 3478c700 -1 osd.5 255 build_incremental_map_msg missing incremental map 255 2019-12-11T22:14:05.713 INFO:teuthology.orchestra.run.smithi079.stderr:testStackedOverwrite (test.functional.tests.TestFile) ... ok 2019-12-11T22:14:05.781 INFO:teuthology.orchestra.run.smithi079.stderr:testTooLongName (test.functional.tests.TestFile) ... ok 2019-12-11T22:14:12.884 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:41:59.815 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:12.936 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:41:59.866 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:12.938 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:41:59.869 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:12.944 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:41:59.874 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:12.994 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:41:59.895 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:12.997 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:41:59.897 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:13.004 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:41:59.935 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:13.010 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:41:59.941 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:13.088 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:41:59.989 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:14:20.459 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:20.452+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:20.956 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:20.949+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:21.443 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:21.438+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:21.595 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:21.587+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:21.830 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:21.821+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:21.938 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:21.930+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:22.480 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:22.475+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:22.592 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:22.584+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:22.811 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:22.803+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:22.961 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:22.952+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:23.396 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:23.389+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:23.522 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:23.517+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:23.638 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:23.629+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:23.839 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:23.832+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:23.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:23.934+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:24.364 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:24.359+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:24.572 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:24.567+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:24.676 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:24.668+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:24.864 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:24.857+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:24.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:24.902+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:24.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:24.966+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:25.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:25.356+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:25.545 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:25.536+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:25.670 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:25.662+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:25.839 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:25.830+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:25.862 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:25.856+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:25.977 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:25.970+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:26.389 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:26.384+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:26.526 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:26.521+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:26.648 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:26.637+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:26.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:26.831+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:26.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:26.832+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:26.865 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:26.856+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:26.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:26.966+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:27.422 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:27.415+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:27.562 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:27.557+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:27.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:27.689+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:27.884 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:27.879+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:27.885 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:27.882+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:27.906 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:27.898+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:28.013 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:27.998+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:28.391 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:28.381+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:28.551 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:28.545+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:28.737 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:28.729+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:28.911 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:28.894+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:28.920 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:28.895+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:28.926 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:28.919+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:28.989 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:28.967+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:29.342 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:29.337+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:29.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:29.566+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:29.768 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:29.761+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:29.909 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:29.902+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:29.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:29.924+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:29.942 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:29.935+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:29.943 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:29.939+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:30.326 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:30.319+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:30.532 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:30.528+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:30.799 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:30.792+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:30.881 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:30.874+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:30.900 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:30.892+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:30.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:30.906+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:30.921 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:30.909+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:31.147 INFO:teuthology.orchestra.run.smithi079.stderr:testZeroByteFile (test.functional.tests.TestFile) ... ok 2019-12-11T22:14:31.295 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:31.286+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:31.524 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:31.507+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:31.832 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:31.826+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:31.896 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:31.887+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:31.907 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:31.900+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:31.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:31.941+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:31.946 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:31.943+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:32.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:32.312+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:32.476 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:32.471+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:32.818 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:32.811+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:32.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:32.845+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:32.866 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:32.859+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:32.955 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:32.950+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:32.963 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:32.951+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:33.273 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:33.267+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:33.300 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:33.294+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:33.451 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:33.447+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:33.814 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:33.799+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:33.831 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:33.824+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:33.885 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:33.870+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:33.950 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:33.944+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:33.953 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:33.948+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:34.276 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:34.269+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:34.311 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:34.306+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:34.472 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:34.467+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:34.788 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:34.779+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:34.864 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:34.858+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:34.877 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:34.870+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:34.951 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:34.947+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:34.953 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:34.950+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:35.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:35.308+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:35.325 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:35.320+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:35.426 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:35.421+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:35.829 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:35.824+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:35.833 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:35.827+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:35.866 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:35.859+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:35.936 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:35.930+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:35.936 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:35.932+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:36.329 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:36.323+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:36.355 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:36.348+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:36.356 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:36.351+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:36.456 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:36.451+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:36.792 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:36.786+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:36.821 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:36.813+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:36.892 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:36.885+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:36.916 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:36.909+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:36.916 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:36.912+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:37.355 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:37.350+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:37.381 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:37.374+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:37.382 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:37.377+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:37.474 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:37.470+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:37.812 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:37.794+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:37.835 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:37.812+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:37.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:37.869+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:37.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:37.871+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:37.930 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:37.922+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:38.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:38.348+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:38.398 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:38.391+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:38.399 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:38.394+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:38.441 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:38.437+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:38.814 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:38.807+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:38.832 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:38.825+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:38.890 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:38.883+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:38.894 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:38.889+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:38.895 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:38.892+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:39.346 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:39.340+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:39.403 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:39.394+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:39.405 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:39.397+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:39.432 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:39.427+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:39.817 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:39.810+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:39.866 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:39.860+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:39.907 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:39.899+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:39.942 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:39.937+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:39.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:39.941+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:40.336 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:40.332+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:40.435 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:40.428+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:40.436 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:40.431+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:40.476 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:40.471+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:40.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:40.793+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:40.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:40.899+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:40.909 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:40.904+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:40.917 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:40.906+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:40.927 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:40.920+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:41.335 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:41.331+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:41.482 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:41.475+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:41.483 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:41.478+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:41.497 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:41.492+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:41.805 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:41.793+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:41.895 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:41.890+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:41.896 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:41.891+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:41.946 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:41.938+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:41.954 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:41.947+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:42.377 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:42.372+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:42.498 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:42.491+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:42.499 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:42.493+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:42.512 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:42.507+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:42.767 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:42.760+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:42.866 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:42.861+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:42.869 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:42.865+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:42.945 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:42.932+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:42.945 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:42.938+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:43.347 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:43.341+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:43.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:43.465+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:43.497 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:43.489+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:43.498 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:43.491+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:43.737 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:43.730+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:43.888 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:43.881+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:43.889 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:43.884+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:43.945 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:43.936+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:43.966 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:43.959+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:44.355 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:44.351+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:44.438 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:44.433+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:44.542 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:44.534+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:44.543 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:44.537+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:44.700 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:44.692+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:44.864 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:44.859+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:44.868 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:44.861+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:44.948 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:44.938+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:44.953 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:44.946+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:45.371 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:45.364+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:45.425 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:45.418+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:45.511 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:45.504+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:45.515 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:45.509+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:45.696 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:45.689+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:45.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:45.876+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:45.885 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:45.878+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:45.948 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:45.940+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:45.976 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:45.969+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:46.341 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:46.334+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:46.418 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:46.410+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:46.562 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:46.555+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:46.566 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:46.560+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:46.693 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:46.686+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:46.853 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:46.847+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:46.853 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:46.848+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:46.947 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:46.939+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:46.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:46.991+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:47.301 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:47.293+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:47.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:47.369+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:47.536 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:47.522+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:47.538 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:47.530+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:47.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:47.666+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:47.868 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:47.863+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:47.870 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:47.865+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:47.957 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:47.950+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:47.961 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:47.954+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:48.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:48.323+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:48.351 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:48.345+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:48.531 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:48.523+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:48.532 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:48.526+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:48.725 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:48.718+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:48.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:48.835+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:48.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:48.836+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:48.953 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:48.945+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:49.005 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:48.987+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:49.292 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:49.286+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:49.346 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:49.340+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:49.543 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:49.535+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:49.545 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:49.538+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:49.696 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:49.688+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:49.836 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:49.830+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:49.839 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:49.834+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:49.983 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:49.975+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:50.010 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:49.997+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:50.308 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:50.303+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:50.379 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:50.371+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:50.595 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:50.588+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:50.599 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:50.593+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:50.677 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:50.669+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:50.800 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:50.793+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:50.800 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:50.795+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:50.957 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:50.951+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:50.998 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:50.991+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:51.300 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:51.292+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:51.377 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:51.371+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:51.619 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:51.610+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:51.620 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:51.613+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:51.638 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:51.630+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:51.798 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:51.793+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:51.801 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:51.794+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:51.976 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:51.968+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:51.994 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:51.986+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:52.259 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:52.253+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:52.395 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:52.389+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:52.621 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:52.614+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:52.622 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:52.616+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:52.676 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:52.669+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:52.797 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:52.792+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:52.801 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:52.797+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:52.996 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:52.987+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:53.010 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:53.002+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:53.265 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:53.259+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:53.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:53.414+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:53.648 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:53.641+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:53.649 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:53.643+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:53.727 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:53.719+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:53.758 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:53.752+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:53.759 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:53.755+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:54.036 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:54.029+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:54.059 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:54.051+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:54.278 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:54.273+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:54.408 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:54.403+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:54.667 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:54.654+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:54.671 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:54.657+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:54.691 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:54.685+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:54.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:54.782+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:54.787 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:54.783+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:55.023 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:55.016+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:55.035 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:55.027+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:55.253 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:55.247+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:55.454 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:55.448+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:55.691 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:55.683+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:55.695 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:55.687+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:55.697 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:55.690+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:55.807 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:55.802+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:55.807 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:55.803+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:55.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:55.991+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:56.028 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:56.020+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:56.230 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:56.225+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:56.422 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:56.417+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:56.650 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:56.640+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:56.695 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:56.686+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:56.696 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:56.689+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:56.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:56.799+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:56.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:56.801+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:57.050 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:57.043+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:57.069 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:57.062+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:57.253 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:57.241+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:57.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:57.401+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:57.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:57.609+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:57.658 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:57.650+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:57.659 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:57.653+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:57.829 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:57.824+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:57.832 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:57.828+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:57.961 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:14:57.954+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e255 v255) 2019-12-11T22:14:58.041 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:58.033+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:58.097 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:58.088+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:58.287 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:58.281+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:58.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:58.413+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:58.585 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:58.578+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:58.648 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:58.641+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:58.651 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:58.645+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:58.803 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:58.796+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:58.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:58.799+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:14:59.002 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:14:58.990+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:14:59.089 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:14:59.081+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:14:59.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:14:59.312+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:14:59.423 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:14:59.416+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:14:59.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:14:59.597+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:14:59.629 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:59.621+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:14:59.634 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:14:59.623+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:14:59.764 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:59.756+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:14:59.774 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:14:59.766+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:00.022 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:00.015+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:00.082 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:00.075+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:00.378 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:00.369+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:00.440 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:00.434+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:00.586 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:00.578+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:00.589 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:00.583+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:00.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:00.597+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:00.790 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:00.784+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:00.791 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:00.786+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:00.978 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:00.971+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:01.069 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:01.061+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:01.412 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:01.407+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:01.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:01.475+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:01.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:01.553+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:01.581 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:01.570+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:01.581 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:01.572+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:01.790 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:01.784+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:01.791 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:01.787+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:02.015 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:02.004+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:02.087 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:02.080+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:02.419 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:02.413+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:02.445 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:02.441+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:02.534 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:02.525+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:02.535 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:02.528+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:02.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:02.582+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:02.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:02.834+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:02.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:02.837+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:02.979 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:15:02.969+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e255 v255) 2019-12-11T22:15:03.017 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:03.006+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:03.078 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:03.071+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:03.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:03.423+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:03.461 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:03.456+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:03.552 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:03.541+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:03.552 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:03.544+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:03.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:03.582+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:03.809 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:03.803+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:03.810 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:03.805+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:03.976 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:03.968+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:04.041 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:04.033+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:04.454 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:04.447+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:04.496 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:04.488+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:04.594 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:04.585+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:04.595 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:04.589+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:04.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:04.587+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:04.829 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:04.820+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:04.829 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:04.824+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:04.993 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:04.985+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:04.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:04.991+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:05.419 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:05.414+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:05.449 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:05.441+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:05.554 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:05.547+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:05.557 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:05.552+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:05.646 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:05.638+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:05.876 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:05.869+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:05.877 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:05.871+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:05.987 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:05.979+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:06.036 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:06.027+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:06.433 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:06.429+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:06.446 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:06.440+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:06.617 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:06.603+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:06.617 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:06.607+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:06.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:06.607+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:06.849 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:06.844+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:06.853 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:06.849+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:06.992 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:06.986+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:07.084 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:07.077+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:07.468 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:15:07.462+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.100179+0000 front 2019-12-11T22:13:37.120184+0000 (oldest deadline 2019-12-11T22:14:19.905749+0000) 2019-12-11T22:15:07.469 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:15:07.462+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:37.736300+0000 front 2019-12-11T22:13:37.775506+0000 (oldest deadline 2019-12-11T22:14:22.413882+0000) 2019-12-11T22:15:07.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:15:07.565+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.620780+0000 front 2019-12-11T22:13:37.099989+0000 (oldest deadline 2019-12-11T22:14:20.661478+0000) 2019-12-11T22:15:07.616 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:07.609+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:13:56.315527+0000 (oldest deadline 2019-12-11T22:14:36.315527+0000) 2019-12-11T22:15:07.616 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:15:07.611+0000 1694f700 -1 osd.5 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e254) 2019-12-11T22:15:07.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:07.842+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:39.198638+0000 front 2019-12-11T22:13:39.255658+0000 (oldest deadline 2019-12-11T22:14:24.377127+0000) 2019-12-11T22:15:07.849 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:07.844+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:08.012 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:15:08.004+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:36.852405+0000 front 2019-12-11T22:13:36.858889+0000 (oldest deadline 2019-12-11T22:14:21.432450+0000) 2019-12-11T22:15:08.041 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:15:08.037+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e255 v255) 2019-12-11T22:15:08.057 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:15:08.050+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:13:38.657831+0000 front 2019-12-11T22:13:38.612621+0000 (oldest deadline 2019-12-11T22:14:20.864722+0000) 2019-12-11T22:15:09.132 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:09.031+0000 1694f700 -1 osd.3 256 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:108203 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e252) 2019-12-11T22:15:10.153 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:15:10.082+0000 1694f700 -1 osd.3 257 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108315 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5492936896 gen 1] snapc 0=[] ondisk+write+known_if_redirected e255) 2019-12-11T22:15:10.369 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:15:10.324+0000 20e25700 -1 RGWWatcher::handle_error cookie 5492936896 err (107) Transport endpoint is not connected 2019-12-11T22:16:18.375 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:44:05.305 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.405 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:44:05.335 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.439 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:44:05.369 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.446 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:44:05.377 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.464 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:44:05.364 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.467 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:44:05.368 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.516 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:44:05.447 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.560 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:44:05.490 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:18.577 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:44:05.478 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:16:34.212 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:34.206+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:34.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:34.316+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:34.729 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:34.719+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:35.206 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:35.200+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:35.312 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:35.307+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:35.373 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:35.368+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:35.754 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:35.750+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:36.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:36.182+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:36.327 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:36.322+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:36.347 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:36.342+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:36.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:36.749+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:37.164 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:37.152+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:37.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:37.202+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:37.355 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:37.341+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:37.373 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:37.361+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:37.773 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:37.768+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:38.116 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:38.111+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:38.217 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:38.210+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:38.370 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:38.365+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:38.384 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:38.380+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:38.733 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:38.728+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:38.829 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:38.822+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:39.141 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:39.136+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:39.262 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:39.256+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:39.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:39.392+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:39.430 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:39.425+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:39.729 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:39.722+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:39.772 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:39.768+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:39.865 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:39.860+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:40.194 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:40.189+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:40.253 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:40.240+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:40.357 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:40.352+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:40.417 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:40.412+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:40.761 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:40.754+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:40.769 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:40.765+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:40.858 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:40.851+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:41.240 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:41.235+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:41.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:41.243+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:41.324 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:41.318+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:41.409 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:41.404+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:41.723 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:41.719+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:41.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:41.794+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:41.890 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:41.882+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:42.268 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:42.262+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:42.276 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:42.271+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:42.320 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:42.314+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:42.418 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:42.414+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:42.756 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:42.751+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:42.818 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:42.802+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:42.857 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:42.851+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:43.258 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:43.251+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:43.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:43.313+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:43.324 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:43.318+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:43.442 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:43.437+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:43.789 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:43.783+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:43.826 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:43.820+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:43.883 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:43.875+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:44.232 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:44.225+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:44.276 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:44.271+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:44.347 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:44.341+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:44.446 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:44.441+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:44.827 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:44.822+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:44.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:44.845+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:44.877 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:44.851+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:45.263 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:45.256+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:45.266 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:45.261+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:45.334 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:45.328+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:45.410 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:45.405+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:45.823 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:45.818+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:45.847 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:45.840+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:45.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:45.900+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:46.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:46.289+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:46.310 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:46.305+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:46.376 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:46.371+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:46.383 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:46.378+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:46.856 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:46.851+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:46.887 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:46.880+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:46.953 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:46.947+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:47.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:47.290+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:47.304 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:47.299+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:47.384 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:47.379+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:47.389 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:47.383+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:47.908 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:47.903+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:47.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:47.930+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:47.985 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:47.979+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:48.293 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:48.287+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:48.305 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:48.300+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:48.380 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:48.375+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:48.401 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:48.395+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:48.879 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:48.874+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:48.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:48.903+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:48.993 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:48.983+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:49.305 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:49.299+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:49.316 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:49.311+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:49.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:49.368+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:49.381 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:49.376+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:49.862 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:49.857+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:49.932 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:49.924+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:50.008 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:50.002+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:50.260 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:50.254+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:50.278 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:50.271+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:50.365 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:50.361+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:50.394 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:50.389+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:50.898 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:50.893+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:50.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:50.926+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:50.996 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:50.977+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:51.232 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:51.225+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:51.302 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:51.297+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:51.355 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:51.349+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:51.357 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:51.352+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:51.889 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:51.885+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:51.913 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:51.906+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:52.027 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:52.020+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:52.269 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:52.263+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:52.314 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:52.308+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:52.340 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:52.335+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:52.347 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:52.337+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:52.891 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:52.884+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:52.892 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:52.888+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:53.080 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:53.072+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:53.302 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:53.297+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:53.318 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:53.312+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:53.359 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:53.354+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:53.376 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:53.370+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:53.924 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:53.917+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:53.927 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:53.922+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:54.105 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:54.095+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:54.294 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:54.289+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:54.333 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:54.327+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:54.335 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:54.330+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:54.354 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:54.348+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:54.916 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:54.910+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:54.939 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:54.932+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:55.121 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:55.114+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:55.288 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:55.281+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:55.306 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:55.297+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:55.337 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:55.333+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:55.340 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:55.334+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:55.914 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:55.909+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:55.918 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:55.912+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:56.162 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:56.149+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:56.283 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:56.277+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:56.315 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:56.310+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:56.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:56.309+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:56.333 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:56.328+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:56.878 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:56.872+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:56.921 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:56.915+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:57.198 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:57.190+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:57.262 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:57.255+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:57.300 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:57.285+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:57.300 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:57.294+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:57.354 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:57.348+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:57.862 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:57.857+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:57.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:57.903+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:58.178 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:58.172+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:58.234 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:58.229+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:58.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:58.248+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:58.281 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:58.276+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:58.344 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:58.338+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:58.819 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:58.813+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:58.891 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:58.884+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:16:59.197 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:16:59.190+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:16:59.228 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:16:59.221+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:16:59.229 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:16:59.223+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:16:59.275 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:16:59.269+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:16:59.338 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:16:59.331+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:16:59.793 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:16:59.787+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:16:59.864 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:16:59.857+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:00.162 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:00.155+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:00.217 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:00.209+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:00.226 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:00.220+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:00.289 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:00.280+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:00.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:00.312+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:00.825 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:00.820+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:00.832 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:00.825+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:01.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:01.150+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:01.198 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:01.192+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:01.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:01.245+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:01.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:01.276+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:01.325 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:01.319+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:01.796 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:01.790+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:01.841 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:01.836+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:02.184 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:02.177+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:02.203 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:02.196+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:02.257 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:02.250+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:02.279 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:02.273+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:02.331 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:02.326+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:02.812 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:02.806+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:02.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:02.844+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:03.145 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:03.139+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:03.190 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:03.183+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:03.272 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:03.266+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:03.313 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:03.308+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:03.358 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:03.353+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:03.839 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:03.834+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:03.844 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:03.838+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:04.122 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:04.117+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:04.181 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:04.174+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:04.289 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:04.282+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:04.321 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:04.316+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:04.353 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:04.348+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:04.846 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:04.840+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:04.883 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:04.879+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:05.112 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:05.107+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:05.143 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:05.137+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:05.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:05.308+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:05.350 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:05.345+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:05.397 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:05.392+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:05.861 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:05.854+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:05.932 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:05.927+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:06.094 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:06.089+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:06.107 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:06.100+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:06.315 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:06.308+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:06.356 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:06.351+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:06.423 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:06.417+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:06.862 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:06.856+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:06.969 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:06.964+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:07.079 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:07.073+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:07.090 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:07.083+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:07.292 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:07.285+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:07.372 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:07.366+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:07.395 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:07.390+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:07.910 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:07.893+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:08.009 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:08.004+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:08.040 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:08.032+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:08.069 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:08.055+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:08.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:08.243+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:08.358 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:08.352+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:08.400 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:08.394+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:08.477 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:17:08.473+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 43sec e260 v260) 2019-12-11T22:17:08.874 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:08.867+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:09.017 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:09.013+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:09.024 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:09.017+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:09.062 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:09.057+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:09.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:09.241+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:09.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:09.347+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:09.379 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:09.374+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:09.928 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:09.921+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:10.032 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:10.025+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:10.050 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:10.031+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:10.063 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:10.058+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:10.236 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:10.230+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:10.314 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:10.308+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:10.420 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:10.415+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:10.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:10.925+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:11.004 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:10.995+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:11.028 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:11.020+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:11.041 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:11.036+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:11.278 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:11.263+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:11.337 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:11.332+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:11.427 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:11.421+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:11.920 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:11.914+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:12.024 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:12.020+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:12.026 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:12.020+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:12.030 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:12.025+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:12.243 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:12.233+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:12.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:12.349+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:12.416 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:12.411+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:12.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:17:12.929+0000 1694f700 -1 osd.7 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.596909+0000 front 2019-12-11T22:15:53.853188+0000 (oldest deadline 2019-12-11T22:16:39.480940+0000) 2019-12-11T22:17:12.996 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:17:12.989+0000 1694f700 -1 osd.1 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:48.916050+0000 front 2019-12-11T22:15:49.033038+0000 (oldest deadline 2019-12-11T22:16:33.838128+0000) 2019-12-11T22:17:13.016 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:17:13.012+0000 1694f700 -1 osd.0 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.180613+0000 front 2019-12-11T22:15:55.324813+0000 (oldest deadline 2019-12-11T22:16:36.360500+0000) 2019-12-11T22:17:13.064 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:17:13.057+0000 1694f700 -1 osd.4 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:52.045443+0000 front 2019-12-11T22:15:52.244240+0000 (oldest deadline 2019-12-11T22:16:37.888609+0000) 2019-12-11T22:17:13.256 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:17:13.250+0000 1694f700 -1 osd.6 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:49.174353+0000 front 2019-12-11T22:15:49.334425+0000 (oldest deadline 2019-12-11T22:16:34.044873+0000) 2019-12-11T22:17:13.365 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:17:13.360+0000 1694f700 -1 osd.2 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:50.609258+0000 front 2019-12-11T22:15:50.786223+0000 (oldest deadline 2019-12-11T22:16:34.107405+0000) 2019-12-11T22:17:13.460 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:17:13.455+0000 1694f700 -1 osd.3 260 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:15:53.171397+0000 front 2019-12-11T22:15:52.784268+0000 (oldest deadline 2019-12-11T22:16:34.699574+0000) 2019-12-11T22:17:13.552 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:17:13.544+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 43sec e260 v260) 2019-12-11T22:17:18.846 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:45:05.777 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.850 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:45:05.781 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.860 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:45:05.791 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.865 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:45:05.796 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.866 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:45:05.766 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.873 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:45:05.774 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.955 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:45:05.886 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.969 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:45:05.870 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:18.972 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:45:05.903 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:17:49.761 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:17:49.733+0000 37792700 -1 osd.5 266 build_incremental_map_msg missing incremental map 266 2019-12-11T22:17:49.787 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:17:49.743+0000 32f89700 -1 osd.5 266 build_incremental_map_msg missing incremental map 266 2019-12-11T22:17:49.961 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:17:49.859+0000 34f8d700 -1 osd.5 266 build_incremental_map_msg missing incremental map 266 2019-12-11T22:17:49.966 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:17:49.868+0000 34f8d700 -1 osd.5 266 build_incremental_map_msg missing incremental map 266 2019-12-11T22:17:49.988 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:17:49.884+0000 31f87700 -1 osd.5 266 build_incremental_map_msg missing incremental map 266 2019-12-11T22:17:50.235 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:17:50.227+0000 34f8d700 -1 osd.5 266 build_incremental_map_msg missing incremental map 266 2019-12-11T22:17:52.060 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:17:52.044+0000 30f85700 -1 osd.5 266 build_incremental_map_msg missing incremental map 266 2019-12-11T22:18:12.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:12.627+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:12.679 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:12.671+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:12.727 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:12.723+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:12.824 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:12.819+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:13.639 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:13.624+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:13.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:13.641+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:13.722 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:13.712+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:13.793 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:13.789+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:13.839 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:13.830+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:14.638 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:14.629+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:14.663 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:14.655+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:14.751 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:14.746+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:14.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:14.753+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:14.875 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:14.867+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:15.527 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:15.521+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:15.653 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:15.644+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:15.653 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:15.645+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:15.779 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:15.774+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:15.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:15.782+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:15.848 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:15.838+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:16.538 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:16.530+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:16.614 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:16.606+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:16.685 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:16.677+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:16.743 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:16.736+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:16.798 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:16.793+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:16.824 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:16.816+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:17.575 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:17.570+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:17.620 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:17.613+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:17.644 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:17.633+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:17.742 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:17.736+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:17.816 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:17.803+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:17.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:17.826+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:17.832 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:17.828+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:18.624 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:18.619+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:18.640 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:18.633+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:18.672 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:18.659+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:18.701 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:18.696+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:18.803 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:18.797+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:18.803 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:18.799+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:18.835 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:18.828+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:19.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:19.593+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:19.636 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:19.629+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:19.692 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:19.683+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:19.721 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:19.715+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:19.827 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:19.819+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:19.850 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:19.844+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:19.852 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:19.847+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:20.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:20.619+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:20.640 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:20.634+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:20.655 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:20.648+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:20.774 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:20.764+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:20.824 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:20.817+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:20.865 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:20.857+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:20.865 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:20.858+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:21.036 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:21.028+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:21.586 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:21.579+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:21.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:21.628+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:21.665 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:21.660+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:21.740 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:21.735+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:21.816 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:21.809+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:21.891 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:21.886+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:21.893 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:21.890+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:21.995 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:21.989+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:22.616 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:22.609+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:22.646 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:22.641+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:22.663 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:22.655+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:22.714 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:22.709+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:22.824 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:22.814+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:22.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:22.927+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:22.932 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:22.929+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:23.050 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:23.040+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:23.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:23.621+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:23.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:23.629+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:23.669 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:23.665+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:23.671 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:23.665+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:23.816 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:23.811+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:23.924 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:23.918+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:23.925 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:23.920+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:24.061 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:24.053+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:24.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:24.601+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:24.649 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:24.643+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:24.709 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:24.703+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:24.711 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:24.704+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:24.809 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:24.797+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:24.930 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:24.922+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:24.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:24.925+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:25.098 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:25.093+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:25.629 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:25.621+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:25.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:25.627+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:25.667 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:25.660+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:25.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:25.702+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:25.837 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:25.831+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:25.947 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:25.941+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:25.949 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:25.944+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:26.132 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:26.126+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:26.136 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:26.131+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:26.608 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:26.601+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:26.681 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:26.674+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:26.684 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:26.678+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:26.702 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:26.696+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:26.809 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:26.802+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:26.988 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:26.980+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:26.988 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:26.983+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:27.156 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:27.149+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:27.161 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:27.156+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:27.639 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:27.633+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:27.671 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:27.663+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:27.709 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:27.702+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:27.734 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:27.728+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:27.765 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:27.759+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:27.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:27.937+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:27.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:27.939+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:28.134 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:28.126+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:28.135 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:28.129+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:28.652 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:28.638+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:28.654 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:28.647+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:28.715 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:28.708+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:28.757 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:28.749+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:28.761 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:28.753+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:28.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:28.936+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:28.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:28.937+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:29.173 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:29.166+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:29.174 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:29.169+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:29.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:29.641+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:29.683 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:29.677+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:29.689 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:29.683+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:29.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:29.735+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:29.801 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:29.793+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:29.984 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:29.976+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:29.985 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:29.979+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:30.205 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:30.185+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:30.205 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:30.188+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:30.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:30.633+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:30.686 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:30.680+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:30.701 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:30.693+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:30.722 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:30.714+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:30.832 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:30.825+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:30.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:30.932+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:30.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:30.933+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:31.245 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:31.238+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:31.249 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:31.244+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:31.666 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:31.658+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:31.672 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:31.664+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:31.695 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:31.690+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:31.717 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:31.710+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:31.805 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:31.797+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:31.939 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:31.934+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:31.940 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:31.936+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:32.244 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:32.236+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:32.244 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:32.238+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:32.651 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:32.638+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:32.718 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:32.710+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:32.720 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:32.715+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:32.721 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:32.716+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:32.799 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:32.791+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:32.900 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:32.895+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:32.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:32.899+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:33.219 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:33.211+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:33.220 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:33.214+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:33.654 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:33.646+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:33.674 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:33.670+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:33.710 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:33.703+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:33.739 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:33.734+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:33.853 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:33.845+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:33.933 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:33.928+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:33.935 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:33.931+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:34.195 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:34.187+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:34.196 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:34.190+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:34.657 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:34.646+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:34.710 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:34.705+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:34.754 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:34.746+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:34.786 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:34.780+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:34.889 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:34.880+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:34.940 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:34.936+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:34.943 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:34.939+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:35.168 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:35.161+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:35.171 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:35.165+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:35.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:35.642+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:35.713 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:35.709+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:35.755 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:35.748+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:35.825 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:35.820+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:35.929 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:35.924+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:35.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:35.925+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:35.935 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:35.927+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:36.151 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:36.144+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:36.154 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:36.149+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:36.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:36.629+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:36.740 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:36.735+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:36.770 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:36.763+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:36.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:36.806+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:36.886 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:36.880+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:36.887 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:36.882+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:36.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:36.899+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:37.131 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:37.124+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:37.133 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:37.126+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:37.661 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:37.654+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:37.712 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:37.708+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:37.745 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:37.739+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:37.815 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:37.810+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:37.876 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:37.869+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:37.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:37.924+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:37.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:37.930+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:38.116 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:38.108+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:38.117 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:38.111+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:38.621 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:38.614+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:38.690 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:38.684+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:38.788 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:38.783+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:38.791 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:38.785+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:38.890 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:38.885+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:38.891 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:38.887+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:38.905 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:38.898+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:39.116 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:39.108+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:39.117 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:39.111+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:39.674 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:39.666+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:39.688 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:39.682+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:39.762 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:39.755+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:39.817 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:39.805+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:39.897 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:39.889+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:39.897 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:39.891+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:39.924 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:39.916+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:40.141 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:40.133+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:40.145 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:40.138+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:40.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:40.632+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:40.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:40.714+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:40.742 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:40.737+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:40.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:40.786+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:40.918 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:40.913+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:40.919 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:40.914+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:40.974 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:40.963+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:41.135 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:41.128+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:41.138 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:41.133+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:41.680 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:41.673+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:41.764 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:41.758+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:41.766 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:41.759+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:41.818 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:41.813+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:41.952 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:41.947+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:41.954 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:41.948+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:42.005 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:41.997+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:42.129 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:42.121+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:42.129 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:42.123+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:42.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:42.700+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:42.766 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:42.758+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:42.815 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:42.810+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:42.836 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:42.832+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:42.948 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:42.926+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:42.948 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:42.929+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:43.029 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:43.021+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:43.162 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:43.154+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:43.164 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:43.157+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:43.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:43.718+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:43.739 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:43.732+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:43.815 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:43.809+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:43.856 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:43.851+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:43.876 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:18:43.871+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e266 v266) 2019-12-11T22:18:43.914 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:43.905+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:43.915 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:43.907+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:44.029 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:44.022+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:44.183 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:44.170+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:44.184 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:44.173+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:44.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:44.714+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:44.750 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:44.742+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:44.825 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:44.820+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:44.866 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:44.861+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:44.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:44.878+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:44.884 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:44.879+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:45.041 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:45.033+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:45.133 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:45.125+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:45.136 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:45.130+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:45.708 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:45.701+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:45.720 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:45.711+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:45.864 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:45.858+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:45.897 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:45.892+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:45.905 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:45.900+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:45.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:45.904+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:46.027 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:46.016+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:46.120 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:46.113+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:46.124 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:46.118+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:46.691 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:46.684+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:46.749 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:46.740+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:46.871 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:46.858+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:46.925 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:46.920+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:46.927 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:46.921+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:46.929 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:46.923+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:47.057 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:47.050+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:47.162 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:47.154+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:47.163 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:47.157+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:47.711 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:47.703+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:47.717 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:47.709+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:47.897 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:47.892+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:47.930 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:47.923+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:47.974 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:47.968+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:47.975 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:47.971+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:48.066 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:48.058+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:48.187 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:48.179+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:48.188 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:48.181+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:48.668 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:48.660+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:48.681 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:48.674+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:48.890 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:18:48.885+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e266 v266) 2019-12-11T22:18:48.922 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:48.917+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:48.930 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:48.926+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:48.955 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:48.951+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:48.956 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:48.952+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:49.071 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:49.064+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:49.204 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:49.196+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:49.205 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:49.199+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:49.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:49.713+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:49.722 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:49.713+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:49.928 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:49.922+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:49.944 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:49.938+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:49.954 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:49.946+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:49.954 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:49.950+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:50.083 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:50.076+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:50.242 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:50.235+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:50.246 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:50.240+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:50.752 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:50.745+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:50.767 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:50.760+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:50.883 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:50.878+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:50.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:50.937+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:50.979 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:50.972+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:50.979 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:50.975+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:51.060 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:51.052+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:51.219 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:51.211+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:51.220 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:51.214+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:51.752 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:51.744+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:51.804 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:51.791+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:51.886 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:51.881+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:51.972 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:51.967+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:52.002 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:51.995+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:52.003 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:51.998+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:52.055 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:52.048+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:52.191 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:52.182+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:52.192 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:52.185+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:52.731 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:52.722+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:52.766 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:52.748+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:52.913 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:52.907+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:52.953 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:52.942+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:52.965 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:52.960+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:52.968 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:52.964+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:53.093 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:53.085+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:53.187 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:53.181+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:53.188 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:53.182+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:53.700 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:53.694+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:53.789 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:53.780+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:53.898 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:18:53.894+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e266 v266) 2019-12-11T22:18:53.951 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:53.947+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:53.969 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:53.953+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:53.996 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:53.991+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:54.008 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:53.992+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:54.129 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:54.121+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:54.147 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:54.138+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:54.148 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:54.141+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:54.753 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:54.747+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:54.839 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:54.831+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:54.916 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:54.912+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:54.971 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:54.966+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:55.039 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:55.031+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:55.040 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:55.033+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:55.157 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:55.146+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:55.158 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:55.151+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:55.178 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:55.167+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:55.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:55.714+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:55.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:55.870+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:55.947 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:55.940+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:56.004 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:55.999+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:56.027 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:56.022+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:56.028 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:56.023+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:56.182 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:56.173+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:56.183 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:56.176+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:56.232 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:56.219+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:56.685 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:56.678+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:56.923 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:56.916+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:56.923 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:56.917+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:56.967 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:56.956+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:57.034 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:57.028+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:57.035 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:57.031+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:57.187 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:57.179+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:57.188 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:57.182+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:57.217 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:57.209+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:57.663 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:57.655+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:57.907 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:57.902+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:57.938 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:57.931+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:58.003 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:57.997+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:58.011 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:58.004+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:58.012 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:58.008+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:18:58.197 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:58.189+0000 1694f700 -1 osd.5 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:17:45.552564+0000 (oldest deadline 2019-12-11T22:18:25.552564+0000) 2019-12-11T22:18:58.200 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:18:58.194+0000 1694f700 -1 osd.5 266 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:108765 7.0 7.5d9261d0 (undecoded) ondisk+retry+read+known_if_redirected e264) 2019-12-11T22:18:58.243 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:18:58.236+0000 1694f700 -1 osd.4 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.996430+0000 front 2019-12-11T22:17:28.976546+0000 (oldest deadline 2019-12-11T22:18:12.943656+0000) 2019-12-11T22:18:58.710 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:18:58.703+0000 1694f700 -1 osd.6 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:28.963587+0000 front 2019-12-11T22:17:28.936512+0000 (oldest deadline 2019-12-11T22:18:12.333216+0000) 2019-12-11T22:18:58.898 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:18:58.892+0000 1694f700 -1 osd.0 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:31.045938+0000 front 2019-12-11T22:17:31.020736+0000 (oldest deadline 2019-12-11T22:18:12.495900+0000) 2019-12-11T22:18:58.967 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:18:58.956+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 45sec e266 v266) 2019-12-11T22:18:58.974 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:18:58.967+0000 1694f700 -1 osd.7 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:26.813300+0000 front 2019-12-11T22:17:26.743261+0000 (oldest deadline 2019-12-11T22:18:12.008369+0000) 2019-12-11T22:18:58.995 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:18:58.988+0000 1694f700 -1 osd.2 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:30.061073+0000 front 2019-12-11T22:17:30.324148+0000 (oldest deadline 2019-12-11T22:18:15.138600+0000) 2019-12-11T22:18:59.007 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:59.000+0000 1694f700 -1 osd.3 266 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:17:27.823682+0000 front 2019-12-11T22:17:27.821090+0000 (oldest deadline 2019-12-11T22:18:12.507644+0000) 2019-12-11T22:18:59.008 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:18:59.003+0000 1694f700 -1 osd.3 266 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:19:00.069 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:19:00.061+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:108791 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5434564800 gen 1] snapc 0=[] ondisk+write+known_if_redirected e263) 2019-12-11T22:19:01.587 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:19:01.560+0000 20e25700 -1 RGWWatcher::handle_error cookie 5434564800 err (107) Transport endpoint is not connected 2019-12-11T22:20:24.829 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:24.823+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:25.467 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:25.461+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:25.854 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:25.848+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:26.407 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:26.403+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:26.515 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:26.508+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:26.672 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:26.667+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:26.901 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:26.894+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:27.379 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:27.375+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:27.387 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:27.382+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:27.516 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:27.509+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:27.688 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:27.683+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:27.707 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:27.700+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:27.888 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:27.876+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:27.907 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:27.900+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:28.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:28.362+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:28.402 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:28.396+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:28.570 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:28.562+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:28.645 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:28.639+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:28.734 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:28.728+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:28.895 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:28.888+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:28.902 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:28.895+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:29.342 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:29.338+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:29.356 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:29.351+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:29.622 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:29.616+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:29.646 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:29.637+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:29.733 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:29.727+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:29.897 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:29.876+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:29.927 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:29.921+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:30.309 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:30.304+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:30.390 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:30.386+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:30.642 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:30.636+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:30.647 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:30.640+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:30.738 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:30.732+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:30.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:30.899+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:30.930 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:30.924+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:31.321 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:31.317+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:31.443 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:31.438+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:31.617 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:31.610+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:31.627 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:31.623+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:31.718 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:31.712+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:31.860 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:31.853+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:31.975 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:31.968+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:32.325 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:32.320+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:32.426 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:32.422+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:32.629 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:32.622+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:32.664 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:32.658+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:32.744 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:32.739+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:32.907 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:32.900+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:33.026 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:33.020+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:33.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:33.362+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:33.414 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:33.409+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:33.679 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:33.674+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:33.700 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:33.695+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:33.773 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:33.769+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:33.933 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:33.927+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:34.048 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:34.040+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:34.364 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:34.358+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:34.426 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:34.420+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:34.667 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:34.662+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:34.672 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:34.665+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:34.799 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:34.795+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:34.972 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:34.965+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:35.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:35.067+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:35.335 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:35.331+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:35.396 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:35.391+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:35.636 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:35.631+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:35.661 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:35.654+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:35.755 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:35.750+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:35.959 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:35.952+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:36.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:36.110+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:36.347 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:36.340+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:36.364 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:36.358+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:36.621 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:36.615+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:36.653 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:36.646+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:36.797 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:36.792+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:36.983 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:36.975+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:37.157 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:37.150+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:37.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:37.380+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:37.400 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:37.381+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:37.657 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:37.650+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:37.696 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:37.689+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:37.815 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:37.809+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:37.960 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:37.953+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:38.145 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:38.134+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:38.349 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:38.343+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:38.424 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:38.418+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:38.649 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:38.643+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:38.655 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:38.649+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:38.795 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:38.790+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:38.935 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:38.928+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:39.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:39.147+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:39.318 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:39.312+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:39.463 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:39.458+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:39.651 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:39.644+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:39.667 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:39.660+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:39.841 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:39.834+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:39.937 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:39.930+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:40.167 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:40.160+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:40.355 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:40.349+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:40.450 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:40.444+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:40.636 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:40.629+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:40.641 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:40.636+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:40.814 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:40.809+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:40.978 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:40.972+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:41.154 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:41.147+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:41.320 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:41.314+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:41.409 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:41.403+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:41.649 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:41.643+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:41.684 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:41.668+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:41.812 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:41.806+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:42.018 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:42.011+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:42.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:42.158+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:42.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:42.331+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:42.396 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:42.391+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:42.657 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:42.652+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:42.667 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:42.661+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:42.826 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:42.821+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:42.985 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:42.978+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:43.179 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:43.172+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:43.320 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:43.315+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:43.433 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:43.429+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:43.688 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:43.681+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:43.699 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:43.694+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:43.868 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:43.864+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:43.994 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:43.988+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:44.138 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:44.131+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:44.318 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:44.313+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:44.487 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:44.481+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:44.660 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:44.653+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:44.751 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:44.746+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:44.835 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:44.827+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:45.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:45.032+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:45.113 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:45.106+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:45.284 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:45.279+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:45.471 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:45.466+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:45.706 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:45.700+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:45.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:45.795+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:45.804 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:45.799+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:46.035 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:46.029+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:46.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:46.068+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:46.321 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:46.316+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:46.491 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:46.486+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:46.718 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:46.711+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:46.811 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:46.806+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:46.841 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:46.834+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:47.021 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:47.015+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:47.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:47.063+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:47.375 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:47.370+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:47.461 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:47.456+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:47.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:47.718+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:47.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:47.772+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:47.880 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:47.875+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:48.063 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:48.056+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:48.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:48.064+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:48.352 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:48.347+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:48.418 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:48.412+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:48.735 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:48.728+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:48.779 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:48.773+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:48.845 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:48.840+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:49.054 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:49.048+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:49.079 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:49.073+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:49.314 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:49.310+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:49.390 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:49.381+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:49.763 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:49.754+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:49.802 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:49.797+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:49.809 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:49.804+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:50.069 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:50.062+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:50.104 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:50.098+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:50.343 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:50.338+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:50.408 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:50.400+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:50.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:50.714+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:50.787 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:50.783+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:50.856 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:50.849+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:51.076 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:51.069+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:51.134 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:51.128+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:51.328 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:51.323+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:51.416 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:51.411+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:51.747 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:51.742+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:51.774 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:51.768+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:51.831 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:51.826+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:52.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:52.101+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:52.136 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:52.130+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:52.349 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:52.344+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:52.443 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:52.439+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:52.739 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:52.734+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:52.789 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:52.784+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:52.818 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:52.812+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:53.120 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:53.113+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:53.176 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:53.169+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:53.330 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:53.325+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:53.452 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:53.448+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:53.697 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:53.692+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:53.796 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:53.791+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:53.838 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:53.832+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:54.097 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:54.090+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:54.144 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:54.137+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:54.377 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:54.371+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:54.429 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:54.424+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:54.747 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:54.742+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:54.818 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:54.812+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:54.878 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:54.872+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:55.122 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:55.112+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:55.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:55.141+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:55.382 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:55.375+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:55.395 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:55.391+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:55.768 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:55.763+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:55.862 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:55.856+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:55.874 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:55.867+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:56.110 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:56.104+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:56.149 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:56.142+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:56.373 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:56.369+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:56.418 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:56.413+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:56.758 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:56.753+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:56.883 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:56.878+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:56.914 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:56.908+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:57.140 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:57.133+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:57.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:57.157+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:57.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:57.325+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:57.416 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:57.411+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:57.809 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:57.805+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:57.871 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:57.865+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:57.871 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:57.865+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:58.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:58.167+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:58.215 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:58.209+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:58.314 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:58.309+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:58.430 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:58.424+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:58.792 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:58.784+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:58.827 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:58.820+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:58.909 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:58.904+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:20:59.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:20:59.150+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:20:59.225 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:20:59.219+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:20:59.284 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:20:59.278+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:20:59.289 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:20:59.284+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e272 v272) 2019-12-11T22:20:59.445 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:20:59.440+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:20:59.757 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:20:59.752+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:20:59.783 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:20:59.777+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:20:59.898 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:20:59.893+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:21:00.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:21:00.143+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:21:00.226 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:21:00.219+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:21:00.305 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:00.300+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:21:00.463 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:00.457+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:21:00.780 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:00.774+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:21:00.833 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:21:00.827+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:21:00.877 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:21:00.872+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:21:01.147 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:21:01.140+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:21:01.275 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:21:01.269+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:21:01.316 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:01.311+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:21:01.490 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:01.485+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:21:01.823 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:01.818+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:21:01.865 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:21:01.857+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:21:01.867 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:21:01.863+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:21:02.119 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:21:02.112+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:21:02.255 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:21:02.248+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:21:02.326 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:02.320+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:21:02.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:02.504+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:21:02.823 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:02.818+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:21:02.837 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:21:02.832+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:21:02.890 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:21:02.883+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:21:03.105 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:21:03.098+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:21:03.242 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:21:03.236+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:21:03.363 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:03.357+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:21:03.522 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:03.517+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:21:03.787 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:03.782+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.827045+0000 front 2019-12-11T22:19:42.212288+0000 (oldest deadline 2019-12-11T22:20:26.830330+0000) 2019-12-11T22:21:03.886 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:21:03.880+0000 1694f700 -1 osd.1 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.957264+0000 front 2019-12-11T22:19:46.521169+0000 (oldest deadline 2019-12-11T22:20:26.361087+0000) 2019-12-11T22:21:03.927 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:21:03.921+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:40.322386+0000 front 2019-12-11T22:19:40.642502+0000 (oldest deadline 2019-12-11T22:20:25.008540+0000) 2019-12-11T22:21:04.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:21:04.150+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:46.683613+0000 front 2019-12-11T22:19:40.679094+0000 (oldest deadline 2019-12-11T22:20:24.166816+0000) 2019-12-11T22:21:04.216 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:21:04.209+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.959992+0000 front 2019-12-11T22:19:46.598504+0000 (oldest deadline 2019-12-11T22:20:27.250688+0000) 2019-12-11T22:21:04.316 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:04.312+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:45.956119+0000 front 2019-12-11T22:19:45.942524+0000 (oldest deadline 2019-12-11T22:20:26.045741+0000) 2019-12-11T22:21:04.330 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:21:04.327+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e272 v272) 2019-12-11T22:21:04.547 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:04.543+0000 1694f700 -1 osd.3 272 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:19:42.650260+0000 front 2019-12-11T22:19:42.682939+0000 (oldest deadline 2019-12-11T22:20:27.173816+0000) 2019-12-11T22:21:15.619 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:49:02.550 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.624 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:49:02.555 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.635 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:49:02.565 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.641 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:49:02.572 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.648 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:49:02.549 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.662 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:49:02.563 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.892 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:49:02.822 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.945 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:49:02.876 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:15.956 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:49:02.857 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:21:38.127 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:38.117+0000 31f87700 -1 osd.0 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.204 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:38.143+0000 31786700 -1 osd.0 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.222 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:38.189+0000 3378a700 -1 osd.2 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.244 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:38.208+0000 31f87700 -1 osd.0 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.362 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:38.325+0000 31786700 -1 osd.0 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.381 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:38.190+0000 3378a700 -1 osd.2 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.388 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:38.206+0000 3378a700 -1 osd.2 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.411 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:38.285+0000 33f8b700 -1 osd.2 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.605 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:21:38.478+0000 3478c700 -1 osd.2 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:38.765 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:21:38.610+0000 3478c700 -1 osd.0 276 build_incremental_map_msg missing incremental map 276 2019-12-11T22:21:41.489 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:21:41.421+0000 36790700 -1 osd.5 278 build_incremental_map_msg missing incremental map 278 2019-12-11T22:21:41.790 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:21:41.668+0000 35f8f700 -1 osd.5 278 build_incremental_map_msg missing incremental map 278 2019-12-11T22:21:42.081 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:21:42.074+0000 31786700 -1 osd.5 278 build_incremental_map_msg missing incremental map 278 2019-12-11T22:21:44.300 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:21:44.196+0000 38794700 -1 osd.5 278 build_incremental_map_msg missing incremental map 278 2019-12-11T22:21:53.857 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:53.853+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:21:54.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:54.874+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:21:55.858 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:55.850+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:21:56.812 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:56.807+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:21:57.843 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:57.839+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:21:58.873 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:58.869+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:21:59.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:21:59.910+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:00.933 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:00.929+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:01.831 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:01.824+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:01.961 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:01.957+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:02.241 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:02.231+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:02.851 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:02.844+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:02.943 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:02.939+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:03.262 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:03.257+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:03.831 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:03.823+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:03.951 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:03.945+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:04.127 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:04.120+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:04.259 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:04.251+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:04.824 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:04.816+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:04.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:04.908+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:05.118 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:05.110+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:05.253 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:05.246+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:05.818 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:05.810+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:05.923 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:05.917+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:05.925 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:05.919+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:06.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:06.158+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:06.267 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:06.261+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:06.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:06.433+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:06.856 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:06.849+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:06.907 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:06.901+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:06.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:06.903+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:07.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:07.176+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:07.226 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:07.220+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:07.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:07.433+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:07.835 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:07.828+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:07.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:07.877+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:07.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:07.878+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:08.158 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:08.150+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:08.181 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:08.175+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:08.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:08.341+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:08.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:08.465+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:08.812 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:08.804+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:08.914 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:08.908+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:08.915 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:08.910+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:09.123 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:09.116+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:09.195 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:09.189+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:09.393 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:09.386+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:09.498 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:09.492+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:09.827 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:09.818+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:09.932 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:09.924+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:09.936 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:09.931+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:10.003 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:09.996+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:10.108 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:10.102+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:10.201 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:10.196+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:10.371 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:10.361+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:10.496 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:10.490+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:10.787 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:10.779+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:10.899 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:10.894+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:10.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:10.900+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:10.967 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:10.960+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:11.126 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:11.119+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:11.174 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:11.169+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:11.325 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:11.318+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:11.547 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:11.542+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:11.837 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:11.830+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:11.936 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:11.930+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:11.936 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:11.932+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:11.986 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:11.975+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:12.133 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:12.126+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:12.187 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:12.180+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:12.296 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:12.290+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:12.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:12.567+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:12.809 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:12.802+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:12.951 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:12.945+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:12.952 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:12.947+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:13.026 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:13.019+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:13.094 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:13.087+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:13.222 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:13.217+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:13.340 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:13.332+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:13.537 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:13.530+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:13.856 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:13.849+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:13.958 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:13.952+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:13.958 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:13.954+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:14.035 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:14.028+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:14.079 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:14.071+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:14.206 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:14.200+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:14.306 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:14.299+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:14.560 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:14.554+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:14.845 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:14.839+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:14.952 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:14.947+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:14.954 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:14.948+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:15.030 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:15.025+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:15.103 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:15.096+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:15.192 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:15.187+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:15.281 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:15.275+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:15.600 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:15.593+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:15.808 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:15.802+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:15.948 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:15.943+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:15.951 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:15.947+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:16.076 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:16.067+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:16.076 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:16.070+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:16.130 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:16.123+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:16.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:16.207+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:16.305 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:16.297+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:16.608 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:16.603+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:16.853 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:16.847+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:16.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:16.899+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:16.904 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:16.901+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:17.045 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:17.037+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:17.045 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:17.040+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:17.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:17.134+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:17.236 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:17.231+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:17.339 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:17.333+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:17.651 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:17.646+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:17.819 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:17.812+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:17.866 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:17.859+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:17.867 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:17.862+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:18.063 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:18.057+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:18.067 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:18.062+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:18.137 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:18.128+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:18.272 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:18.267+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:18.342 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:18.336+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:18.644 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:18.638+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:18.798 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:18.790+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:18.911 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:18.905+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:18.912 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:18.908+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:19.039 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:19.033+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:19.040 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:19.035+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:19.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:19.135+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:19.301 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:19.297+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:19.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:19.312+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:19.652 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:19.646+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:19.774 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:19.768+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:19.877 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:19.872+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:19.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:19.874+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:20.001 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:19.995+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:20.008 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:20.000+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:20.177 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:20.170+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:20.317 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:20.310+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:20.319 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:20.313+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:20.691 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:20.686+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:20.819 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:20.811+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:20.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:20.870+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:20.877 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:20.873+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:20.981 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:20.975+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:20.985 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:20.980+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:21.218 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:21.212+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:21.288 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:21.282+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:21.369 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:21.363+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:21.660 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:21.654+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:21.856 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:21.849+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:21.881 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:21.877+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:21.882 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:21.878+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:21.980 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:21.974+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:21.981 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:21.976+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:22.187 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:22.181+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:22.275 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:22.270+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:22.334 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:22.327+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:22.629 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:22.623+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:22.902 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:22.894+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:22.917 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:22.912+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:22.919 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:22.915+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:22.942 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:22.934+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:22.943 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:22.936+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:23.205 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:23.197+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:23.239 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:23.234+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:23.387 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:23.373+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:23.610 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:23.604+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:23.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:23.896+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:23.934 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:23.928+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:23.935 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:23.931+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:23.991 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:23.982+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:23.991 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:23.985+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:24.168 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:24.160+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:24.195 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:24.189+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:24.432 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:24.421+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:24.617 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:24.610+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:24.873 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:24.865+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:24.949 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:24.942+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:24.950 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:24.945+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:24.960 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:24.953+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:24.964 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:24.958+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:25.158 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:25.150+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:25.175 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:25.168+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:25.422 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:25.414+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:25.646 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:25.642+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:25.857 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:25.845+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:25.920 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:25.915+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:25.929 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:25.919+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:25.945 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:25.926+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:25.945 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:25.928+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:26.135 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:26.127+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:26.214 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:26.209+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:26.408 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:26.397+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:26.628 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:26.622+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:26.857 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:26.849+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:26.941 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:26.935+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:26.942 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:26.936+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:26.979 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:26.971+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:26.979 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:26.973+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:27.125 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:27.115+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:27.258 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:27.253+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:27.415 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:27.408+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:27.673 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:27.668+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:27.885 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:27.878+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:27.941 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:27.932+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:27.942 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:27.935+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:27.951 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:27.937+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:27.952 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:27.943+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:28.124 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:28.116+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:28.280 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:28.275+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:28.433 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:28.425+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:28.666 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:28.662+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:28.896 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:28.889+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:28.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:28.931+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:28.938 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:28.932+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:28.942 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:28.934+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:28.945 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:28.937+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:29.096 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:29.089+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:29.313 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:29.307+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:29.387 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:29.380+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:29.687 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:29.679+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:29.933 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:29.928+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:29.935 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:29.932+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:29.945 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:29.938+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:29.992 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:29.984+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:29.999 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:29.994+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:30.140 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:30.132+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:30.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:30.341+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:30.355 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:30.349+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:30.726 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:30.721+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:30.928 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:30.923+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:30.929 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:30.925+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:30.999 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:30.985+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:30.999 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:30.987+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:30.999 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:30.989+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:31.137 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:31.128+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:31.382 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:31.376+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:31.399 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:31.392+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:31.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:31.772+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:31.963 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:31.956+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:31.966 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:31.962+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:32.021 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:32.007+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:32.022 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:32.008+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:32.022 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:32.014+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:32.142 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:32.134+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:32.389 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:32.383+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:32.440 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:32.434+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:32.827 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:32.822+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:32.978 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:32.973+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:32.981 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:32.977+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:32.996 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:32.988+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:32.997 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:32.991+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:33.060 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:33.054+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:33.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:33.157+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:33.359 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:33.354+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:33.466 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:33.457+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:33.822 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:33.817+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:34.006 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:34.000+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:34.007 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:34.003+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:34.009 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:34.002+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:34.012 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:34.007+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:34.069 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:34.062+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:34.217 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:34.210+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:34.369 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:34.364+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:34.431 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:34.423+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:34.589 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:22:34.583+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e278 v278) 2019-12-11T22:22:34.838 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:34.833+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:34.967 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:34.960+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:34.970 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:34.965+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:34.990 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:34.984+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:34.990 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:34.987+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:35.076 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:35.069+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:35.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:35.177+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:35.359 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:35.353+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:35.455 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:35.447+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:35.880 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:35.875+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:35.966 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:35.959+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:35.970 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:35.964+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:36.014 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:36.006+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:36.018 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:36.014+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:36.104 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:36.097+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:36.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:36.139+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:36.328 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:36.322+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:36.448 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:36.441+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:36.908 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:36.902+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:36.924 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:36.916+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:36.925 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:36.919+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:37.060 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:37.052+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:37.061 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:37.056+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:37.141 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:37.134+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:37.144 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:37.136+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:37.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:37.347+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:37.485 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:37.478+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:37.884 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:37.880+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:37.948 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:37.940+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:37.949 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:37.943+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:38.018 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:38.012+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:38.018 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:38.014+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:38.103 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:38.095+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:38.154 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:38.146+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:38.310 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:38.304+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:38.465 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:38.458+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:38.872 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:38.866+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:38.961 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:38.953+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:38.963 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:38.957+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:39.028 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:39.023+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.982494+0000 front 2019-12-11T22:21:19.814070+0000 (oldest deadline 2019-12-11T22:22:05.053686+0000) 2019-12-11T22:22:39.032 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:39.028+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:39.105 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:39.097+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:39.136 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:39.122+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:39.323 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:39.317+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:39.458 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:39.450+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:39.606 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:22:39.602+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e278 v278) 2019-12-11T22:22:39.894 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:39.888+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:39.926 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:39.916+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:39.937 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:39.919+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:40.028 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:40.024+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:40.086 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:40.079+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:40.114 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:40.102+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:40.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:40.305+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:40.481 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:40.474+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:40.886 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:40.881+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:40.961 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:40.952+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:40.962 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:40.956+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:41.079 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:41.074+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:41.133 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:41.125+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:41.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:41.166+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:41.339 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:41.333+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:41.524 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:41.517+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:41.926 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:41.921+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:41.989 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:41.980+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:41.991 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:41.982+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:42.045 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:42.040+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:42.131 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:42.123+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:42.220 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:42.213+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:42.330 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:42.325+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:42.574 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:42.566+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:42.975 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:42.969+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:43.009 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:43.002+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:43.009 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:43.004+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:43.082 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:43.078+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:43.134 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:43.127+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:43.274 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:43.266+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:43.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:43.347+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:43.620 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:43.614+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:43.972 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:22:43.967+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:19.994624+0000 front 2019-12-11T22:21:20.101909+0000 (oldest deadline 2019-12-11T22:22:05.816092+0000) 2019-12-11T22:22:43.996 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:43.989+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:21:35.707521+0000 (oldest deadline 2019-12-11T22:22:15.707521+0000) 2019-12-11T22:22:44.000 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:43.994+0000 1694f700 -1 osd.5 278 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-11T22:22:44.101 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:44.097+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:44.164 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:22:44.156+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:15.882229+0000 front 2019-12-11T22:21:15.884556+0000 (oldest deadline 2019-12-11T22:22:01.158410+0000) 2019-12-11T22:22:44.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:22:44.261+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.704507+0000 front 2019-12-11T22:21:20.276385+0000 (oldest deadline 2019-12-11T22:22:03.725178+0000) 2019-12-11T22:22:44.304 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:22:44.300+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:20.286020+0000 front 2019-12-11T22:21:20.790880+0000 (oldest deadline 2019-12-11T22:22:01.949902+0000) 2019-12-11T22:22:44.663 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:22:44.655+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:21:17.222891+0000 front 2019-12-11T22:21:17.262382+0000 (oldest deadline 2019-12-11T22:22:08.077438+0000) 2019-12-11T22:22:44.711 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:22:44.675+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e278 v278) 2019-12-11T22:22:45.079 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:22:45.068+0000 34f8d700 -1 osd.5 279 build_incremental_map_msg missing incremental map 278 2019-12-11T22:22:45.224 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:45.153+0000 1694f700 -1 osd.3 279 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:46.352 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:22:46.201+0000 1694f700 -1 osd.3 280 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:109144 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 385198352 gen 1] snapc 0=[] ondisk+write+known_if_redirected e274) 2019-12-11T22:22:47.097 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:22:47.087+0000 20e25700 -1 RGWWatcher::handle_error cookie 385198352 err (107) Transport endpoint is not connected 2019-12-11T22:23:18.412 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:23:18.390+0000 31f87700 -1 osd.1 283 build_incremental_map_msg missing incremental map 283 2019-12-11T22:24:17.613 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:17.607+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:18.054 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:18.047+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:18.110 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:18.105+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:18.185 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:18.180+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:18.653 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:18.638+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:18.834 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:18.828+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:19.024 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:19.017+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:19.150 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:19.145+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:19.159 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:19.154+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:19.524 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:19.518+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:19.627 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:19.621+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:19.835 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:19.829+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:19.981 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:19.975+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:20.119 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:20.115+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:20.189 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:20.183+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:20.547 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:20.540+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:20.639 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:20.632+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:20.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:20.881+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:21.013 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:21.006+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:21.043 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:21.036+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:21.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:21.125+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:21.187 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:21.180+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:21.560 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:21.552+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:21.607 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:21.601+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:21.859 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:21.852+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:22.032 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:22.026+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:22.077 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:22.070+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:22.112 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:22.106+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:22.182 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:22.178+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:22.597 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:22.592+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:22.618 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:22.610+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:22.818 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:22.812+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:23.046 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:23.039+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:23.083 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:23.078+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:23.163 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:23.150+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:23.187 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:23.183+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:23.608 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:23.600+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:23.643 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:23.637+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:23.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:23.845+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:24.024 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:24.018+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:24.112 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:24.106+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:24.137 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:24.132+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:24.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:24.230+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:24.623 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:24.618+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:24.656 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:24.649+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:24.831 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:24.824+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:25.011 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:25.005+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:25.129 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:25.117+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:25.148 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:25.143+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:25.281 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:25.276+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:25.621 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:25.616+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:25.707 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:25.700+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:25.791 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:25.785+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:26.037 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:26.029+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:26.120 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:26.114+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:26.138 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:26.132+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:26.335 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:26.329+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:26.596 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:26.590+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:26.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:26.679+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:26.755 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:26.749+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:27.087 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:27.079+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:27.140 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:27.136+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:27.156 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:27.151+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:27.367 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:27.362+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:27.586 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:27.581+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:27.719 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:27.712+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:27.730 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:27.723+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:28.121 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:28.114+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:28.152 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:28.146+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:28.211 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:28.206+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:28.324 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:28.317+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:28.573 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:28.568+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:28.692 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:28.685+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:28.753 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:28.746+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:29.110 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:29.104+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:29.112 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:29.106+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:29.235 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:29.230+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:29.362 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:29.357+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:29.574 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:29.569+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:29.662 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:29.655+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:29.719 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:29.712+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:30.082 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:30.077+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:30.140 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:30.134+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:30.276 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:30.272+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:30.375 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:30.370+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:30.625 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:30.620+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:30.677 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:30.669+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:30.745 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:30.732+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:31.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:31.044+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:31.168 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:31.161+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:31.257 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:31.252+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:31.332 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:31.327+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:31.629 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:31.624+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:31.651 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:31.643+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:31.706 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:31.699+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:32.041 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:32.036+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:32.162 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:32.156+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:32.293 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:32.288+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:32.311 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:32.304+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:32.605 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:32.600+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:32.691 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:32.685+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:32.750 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:32.743+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:33.027 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:33.022+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:33.130 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:33.123+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:33.288 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:33.283+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:33.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:33.308+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:33.631 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:33.626+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:33.689 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:33.680+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:33.741 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:33.734+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:34.054 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:34.050+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:34.146 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:34.140+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:34.247 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:34.243+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:34.269 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:34.262+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:34.644 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:34.639+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:34.679 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:34.672+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:34.699 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:34.693+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:35.008 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:35.004+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:35.113 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:35.106+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:35.223 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:35.218+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:35.266 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:35.260+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:35.680 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:35.675+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:35.681 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:35.676+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:35.723 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:35.717+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:35.975 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:35.969+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:36.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:36.110+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:36.179 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:36.174+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:36.294 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:36.288+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:36.670 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:36.664+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:36.701 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:36.694+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:36.762 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:36.755+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:37.000 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:36.995+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:37.138 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:37.132+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:37.165 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:37.161+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:37.302 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:37.297+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:37.658 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:37.653+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:37.713 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:37.706+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:37.751 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:37.744+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:38.030 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:38.024+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:38.156 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:38.149+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:38.185 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:38.177+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:38.291 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:38.287+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:38.709 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:38.703+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:38.728 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:38.721+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:38.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:38.737+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:39.078 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:39.073+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:39.116 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:39.111+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:39.180 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:39.174+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:39.264 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:39.258+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:39.702 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:39.695+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:39.703 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:39.695+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:39.708 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:39.702+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:40.074 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:40.070+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:40.118 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:40.113+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:40.215 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:40.208+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:40.233 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:40.229+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:40.246 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:52:27.177 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.294 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:52:27.225 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.301 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:52:27.232 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.304 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:52:27.235 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.305 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:52:27.206 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.307 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:52:27.209 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.473 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:52:27.403 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.478 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:52:27.409 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.601 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:52:27.502 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:24:40.679 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:40.673+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:40.704 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:40.700+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:40.707 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:40.700+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:41.104 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:41.100+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:41.153 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:41.149+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:41.208 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:41.202+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:41.252 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:41.247+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:41.690 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:41.683+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:41.712 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:41.705+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:41.751 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:41.747+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:42.134 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:42.129+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:42.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:42.181+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:42.190 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:42.185+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:42.274 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:42.268+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:42.699 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:42.693+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:42.740 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:42.735+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:42.741 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:42.734+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:43.142 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:43.138+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:43.187 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:43.182+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:43.202 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:43.195+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:43.258 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:43.252+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:43.695 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:43.689+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:43.698 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:43.693+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:43.792 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:43.786+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:44.106 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:44.102+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:44.177 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:44.171+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:44.211 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:44.206+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:44.257 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:44.251+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:44.678 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:44.672+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:44.722 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:44.717+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:44.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:44.740+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:45.085 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:45.080+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:45.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:45.176+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:45.230 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:45.225+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:45.309 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:45.304+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:45.691 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:45.685+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:45.721 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:45.715+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:45.799 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:45.793+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:46.078 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:46.073+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:46.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:46.131+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:46.213 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:46.207+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:46.310 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:46.305+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:46.654 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:46.647+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:46.753 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:46.748+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:46.828 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:46.821+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:47.102 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:47.096+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:47.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:47.169+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:47.214 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:47.209+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:47.324 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:47.319+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:47.649 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:47.642+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:47.707 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:47.702+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:47.858 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:47.852+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:48.104 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:48.099+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:48.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:48.176+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:48.260 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:48.255+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:48.300 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:48.295+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:48.652 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:48.645+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:48.737 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:48.732+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:48.857 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:48.850+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:49.113 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:49.107+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:49.145 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:49.139+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:49.222 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:49.218+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:49.312 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:49.304+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:49.611 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:49.605+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:49.746 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:49.740+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:49.860 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:49.854+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:50.145 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:50.138+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:50.162 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:50.157+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:50.188 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:50.183+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:50.255 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:24:50.251+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 42sec e283 v283) 2019-12-11T22:24:50.288 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:50.283+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:50.627 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:50.619+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:50.795 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:50.790+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:50.832 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:50.827+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:51.172 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:51.165+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:51.193 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:51.188+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:51.199 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:51.195+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:51.336 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:51.331+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:51.590 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:51.584+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:51.781 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:51.776+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:51.823 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:51.817+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:52.226 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:52.219+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:52.247 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:52.242+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:52.254 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:52.249+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:52.374 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:52.369+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:52.614 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:52.608+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:52.748 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:52.742+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:52.847 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:52.841+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:53.247 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:53.236+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:53.252 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:53.246+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:53.289 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:53.283+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:53.397 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:53.392+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:53.627 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:53.620+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:53.792 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:53.787+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:53.865 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:53.857+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:54.282 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:54.273+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:54.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:54.290+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:54.304 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:54.299+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:54.357 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:54.352+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:24:54.644 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:24:54.637+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.638227+0000 front 2019-12-11T22:23:35.195998+0000 (oldest deadline 2019-12-11T22:24:17.539305+0000) 2019-12-11T22:24:54.750 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:24:54.746+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:37.184128+0000 front 2019-12-11T22:23:36.284174+0000 (oldest deadline 2019-12-11T22:24:18.488833+0000) 2019-12-11T22:24:54.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:24:54.903+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:32.943628+0000 front 2019-12-11T22:23:33.738579+0000 (oldest deadline 2019-12-11T22:24:18.162296+0000) 2019-12-11T22:24:55.271 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:24:55.264+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:35.067435+0000 front 2019-12-11T22:23:36.260011+0000 (oldest deadline 2019-12-11T22:24:20.786951+0000) 2019-12-11T22:24:55.299 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:24:55.295+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:33.183992+0000 front 2019-12-11T22:23:32.924907+0000 (oldest deadline 2019-12-11T22:24:17.374945+0000) 2019-12-11T22:24:55.306 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:24:55.302+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:34.143734+0000 front 2019-12-11T22:23:33.860634+0000 (oldest deadline 2019-12-11T22:24:17.350046+0000) 2019-12-11T22:24:55.308 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:24:55.299+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 42sec e283 v283) 2019-12-11T22:24:55.334 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:24:55.329+0000 1694f700 -1 osd.3 283 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:23:36.259048+0000 front 2019-12-11T22:23:35.847124+0000 (oldest deadline 2019-12-11T22:24:17.536114+0000) 2019-12-11T22:25:09.062 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:52:55.993 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.068 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:52:55.999 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.090 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:52:55.991 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.099 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:52:56.000 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.118 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:52:56.049 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.125 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:52:56.056 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.127 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:52:56.058 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.187 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:52:56.118 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:09.189 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:52:56.090 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:25:28.284 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:28.269+0000 124b0700 -1 osd.0 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.480 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:28.354+0000 31f87700 -1 osd.0 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.534 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:28.463+0000 32387700 -1 osd.7 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.602 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:28.360+0000 31f87700 -1 osd.0 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.639 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:28.485+0000 31f87700 -1 osd.0 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.668 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:28.530+0000 3478c700 -1 osd.0 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.729 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:28.509+0000 3638f700 -1 osd.7 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.779 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:28.769+0000 34f8d700 -1 osd.6 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.806 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:28.569+0000 32387700 -1 osd.7 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.820 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:28.784+0000 35f8f700 -1 osd.6 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.822 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:28.640+0000 3438b700 -1 osd.7 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.835 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:28.785+0000 35f8f700 -1 osd.6 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.859 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:28.639+0000 114ae700 -1 osd.0 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:28.915 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:28.708+0000 3438b700 -1 osd.7 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:29.085 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:28.901+0000 33389700 -1 osd.7 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:29.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:29.039+0000 37f93700 -1 osd.6 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:29.267 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:29.131+0000 38794700 -1 osd.6 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:29.873 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:25:29.562+0000 33f8b700 -1 osd.1 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:29.900 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:25:29.634+0000 32f89700 -1 osd.1 287 build_incremental_map_msg missing incremental map 287 2019-12-11T22:25:32.652 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:25:32.551+0000 31f87700 -1 osd.5 289 build_incremental_map_msg missing incremental map 289 2019-12-11T22:25:34.414 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:25:34.364+0000 38794700 -1 osd.5 289 build_incremental_map_msg missing incremental map 289 2019-12-11T22:25:45.992 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:45.986+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:46.994 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:46.989+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:48.024 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:48.019+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:48.997 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:48.992+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:50.005 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:49.993+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:50.987 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:50.978+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:52.004 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:52.000+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:52.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:52.938+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:25:53.026 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:53.022+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:53.103 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:25:53.095+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:25:54.001 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:53.988+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:25:54.026 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:54.018+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:54.092 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:25:54.079+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:25:54.757 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:54.751+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:25:54.979 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:54.974+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:54.994 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:54.987+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:25:55.041 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:25:55.034+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:25:55.327 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:25:55.322+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:25:55.754 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:55.748+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:25:55.972 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:55.965+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:25:55.973 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:55.969+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:56.043 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:56.034+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:25:56.090 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:25:56.083+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:25:56.368 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:25:56.362+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:25:56.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:56.704+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:25:57.008 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:57.004+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:25:57.010 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:57.006+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:57.097 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:25:57.086+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:25:57.097 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:57.088+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:25:57.234 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:57.225+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:25:57.398 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:25:57.393+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:25:57.664 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:57.658+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:25:58.056 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:58.051+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:25:58.060 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:58.056+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:58.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:25:58.058+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:25:58.087 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:58.078+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:25:58.233 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:58.225+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:25:58.423 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:25:58.419+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:25:58.619 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:58.613+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:25:59.045 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:59.040+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:25:59.047 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:25:59.043+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:25:59.073 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:25:59.065+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:25:59.135 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:25:59.128+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:25:59.215 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:25:59.208+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:25:59.459 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:25:59.454+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:25:59.597 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:25:59.592+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:00.067 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:00.062+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:00.067 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:00.063+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:00.081 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:00.074+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:00.172 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:00.164+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:00.258 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:00.253+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:00.269 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:00.261+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:00.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:00.463+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:00.585 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:00.581+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:01.049 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:01.041+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:01.108 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:01.103+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:01.110 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:01.107+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:01.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:01.215+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:01.238 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:01.232+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:01.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:01.309+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:01.449 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:01.441+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:01.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:01.593+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:02.073 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:02.059+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:02.086 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:02.082+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:02.091 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:02.086+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:02.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:02.239+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:02.267 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:02.260+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:02.280 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:02.270+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:02.408 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:02.403+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:02.590 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:02.585+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:03.066 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:03.059+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:03.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:03.097+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:03.104 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:03.099+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:03.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:03.242+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:03.289 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:03.281+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:03.299 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:03.293+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:03.391 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:03.386+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:03.543 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:03.537+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:04.084 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:04.076+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:04.095 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:04.090+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:04.097 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:04.093+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:04.294 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:04.284+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:04.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:04.290+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:04.354 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:04.347+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:04.399 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:04.395+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:04.583 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:04.578+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:05.081 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:05.074+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:05.123 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:05.118+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:05.123 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:05.119+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:05.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:05.261+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:05.329 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:05.315+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:05.380 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:05.373+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:05.429 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:05.420+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:05.549 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:05.535+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:06.074 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:06.066+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:06.097 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:06.092+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:06.098 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:06.094+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:06.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:06.245+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:06.322 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:06.314+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:06.348 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:06.342+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:06.416 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:06.410+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:06.531 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:06.525+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:07.076 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:07.069+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:07.125 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:07.120+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:07.127 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:07.124+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:07.277 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:07.270+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:07.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:07.281+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:07.399 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:07.392+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:07.403 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:07.397+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:07.408 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:07.403+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:07.583 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:07.576+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:08.040 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:08.031+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:08.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:08.114+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:08.124 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:08.117+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:08.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:08.261+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:08.279 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:08.272+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:08.379 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:08.372+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:08.380 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:08.375+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:08.443 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:08.437+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:08.634 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:08.629+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:09.058 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:09.051+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:09.101 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:09.097+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:09.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:09.098+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:09.286 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:09.279+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:09.293 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:09.286+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:09.417 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:09.409+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:09.418 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:09.412+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:09.456 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:09.451+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:09.644 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:09.638+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:10.110 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:10.105+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:10.111 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:10.108+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:10.112 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:10.106+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:10.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:10.242+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:10.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:10.256+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:10.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:10.458+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:10.465 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:10.458+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:10.466 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:10.461+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:10.676 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:10.671+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:11.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:11.098+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:11.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:11.100+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:11.104 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:11.099+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:11.267 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:11.260+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:11.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:11.265+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:11.440 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:11.433+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:11.441 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:11.435+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:11.520 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:11.512+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:11.711 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:11.706+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:12.089 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:12.082+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:12.109 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:12.104+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:12.113 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:12.108+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:12.283 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:12.277+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:12.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:12.313+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:12.464 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:12.457+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:12.465 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:12.460+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:12.484 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:12.478+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:12.705 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:12.700+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:13.078 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:13.073+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:13.078 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:13.074+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:13.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:13.120+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:13.274 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:13.268+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:13.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:13.274+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:13.464 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:13.458+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:13.466 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:13.459+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:13.467 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:13.462+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:13.677 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:13.672+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:14.073 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:14.067+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:14.073 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:14.069+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:14.115 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:14.109+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:14.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:14.321+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:14.338 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:14.332+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:14.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:14.415+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:14.461 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:14.454+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:14.462 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:14.457+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:14.677 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:14.672+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:15.072 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:15.066+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:15.085 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:15.080+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:15.085 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:15.081+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:15.286 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:15.280+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:15.367 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:15.359+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:15.446 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:15.441+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:15.498 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:15.490+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:15.499 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:15.493+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:15.636 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:15.631+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:16.028 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:16.020+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:16.041 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:16.035+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:16.041 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:16.037+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:16.330 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:16.323+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:16.336 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:16.329+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:16.444 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:16.438+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:16.536 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:16.528+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:16.537 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:16.531+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:16.656 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:16.641+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:17.052 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:17.043+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:17.076 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:17.068+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:17.076 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:17.071+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:17.362 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:17.354+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:17.379 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:17.371+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:17.475 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:17.470+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:17.521 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:17.492+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:17.526 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:17.513+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:17.605 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:17.599+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:18.079 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:18.071+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:18.125 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:18.120+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:18.126 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:18.122+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:18.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:18.365+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:18.380 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:18.373+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:18.438 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:18.434+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:18.524 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:18.517+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:18.531 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:18.519+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:18.585 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:18.581+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:19.111 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:19.105+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:19.136 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:19.131+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:19.136 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:19.132+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:19.337 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:19.330+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:19.400 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:19.394+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:19.405 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:19.398+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:19.501 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:19.494+0000 1694f700 -1 osd.5 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:25:26.680571+0000 (oldest deadline 2019-12-11T22:26:06.680571+0000) 2019-12-11T22:26:19.505 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:19.499+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:19.567 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:19.563+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:20.130 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:20.123+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:20.169 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:20.164+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:20.171 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:20.167+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:20.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:20.354+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:20.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:20.396+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:20.414 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:20.409+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:20.475 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:20.467+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:20.605 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:20.598+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:21.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:21.150+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:21.159 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:21.154+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:21.160 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:21.156+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:21.384 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:21.380+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:21.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:21.402+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:21.413 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:21.405+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:21.525 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:21.515+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:21.621 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:21.610+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:22.172 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:22.167+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:22.173 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:22.168+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:22.191 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:22.184+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:22.374 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:22.365+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:22.374 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:22.367+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:22.413 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:22.407+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:22.573 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:22.567+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:22.589 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:22.584+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:23.226 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:23.220+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:23.230 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:23.225+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:23.232 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:23.225+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:23.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:23.329+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:23.378 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:23.372+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:23.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:23.375+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:23.582 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:23.576+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:23.638 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:23.634+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:24.198 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:24.193+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:24.199 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:24.195+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:24.277 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:24.269+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:24.294 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:24.286+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:24.334 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:24.327+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:24.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:24.336+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:24.547 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:24.540+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:24.620 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:24.615+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:25.210 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:25.203+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:25.211 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:25.206+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:25.250 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:25.243+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:25.277 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:25.269+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:25.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:25.309+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:25.380 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:25.376+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:25.547 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:25.540+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:25.617 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:26:25.610+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 47sec e289 v289) 2019-12-11T22:26:25.635 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:25.630+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:26.225 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:26.219+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:26.230 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:26.225+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:26.281 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:26.274+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:26.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:26.280+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:26.293 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:26.287+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:26.344 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:26.340+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:26.564 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:26.557+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:26.678 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:26.672+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:27.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:27.254+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:27.270 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:27.265+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:27.272 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:27.268+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:27.300 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:27.292+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:27.304 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:27.295+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:27.365 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:27.360+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:27.597 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:27.591+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:27.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:27.692+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:28.282 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:28.276+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:28.283 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:28.278+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:28.303 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:28.295+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:28.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:28.296+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:28.306 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:28.300+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:28.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:28.316+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:28.604 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:28.597+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:28.745 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:28.738+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:29.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:29.275+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:29.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:29.279+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:29.293 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:29.285+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:29.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:29.309+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:29.334 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:29.328+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:29.341 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:29.333+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:29.611 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:29.604+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:29.717 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:29.712+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:30.258 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:30.251+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:30.268 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:30.262+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:30.278 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:30.272+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:30.307 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:30.302+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:30.310 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:30.305+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:30.312 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:30.305+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:30.635 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:26:30.629+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 47sec e289 v289) 2019-12-11T22:26:30.659 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:30.653+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:30.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:30.730+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:31.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:31.217+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:31.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:31.270+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:31.281 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:31.274+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:31.306 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:31.299+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:31.326 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:31.320+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:31.329 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:31.324+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:31.710 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:31.704+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:31.776 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:31.769+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:32.193 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:32.185+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:32.263 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:32.256+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:32.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:32.263+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:32.272 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:32.265+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:32.336 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:32.328+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:32.338 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:32.332+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:32.725 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:32.719+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:32.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:32.808+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:33.221 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:33.213+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:33.246 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:33.240+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:33.274 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:33.260+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:33.317 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:33.305+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:33.336 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:33.326+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:33.337 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:33.328+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:33.713 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:33.707+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:33.800 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:33.796+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:34.198 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:34.191+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:34.260 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:34.252+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:34.273 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:34.268+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:34.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:34.343+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:34.369 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:34.365+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:34.370 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:34.366+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:34.720 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:34.714+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:34.845 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:34.840+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:35.166 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:35.160+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:35.229 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:35.213+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:35.240 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:35.233+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:35.353 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:35.337+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:35.364 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:35.359+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:35.367 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:35.363+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:35.654 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:26:35.648+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 47sec e289 v289) 2019-12-11T22:26:35.685 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:35.679+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:35.844 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:35.838+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:36.191 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:36.182+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:36.209 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:36.203+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:36.271 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:36.262+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:36.368 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:36.358+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:36.368 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:36.362+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:36.375 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:36.368+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:36.723 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:36.713+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:36.886 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:36.877+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:37.192 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:37.182+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:37.240 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:37.236+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:37.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:37.283+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:37.344 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:37.338+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:37.345 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:37.341+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:37.416 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:37.408+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:37.772 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:37.765+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:37.914 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:37.909+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:38.221 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:38.213+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:38.293 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:38.283+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:38.302 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:38.296+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:38.304 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:38.299+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:38.313 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:38.306+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:38.387 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:38.379+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:38.769 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:38.763+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:38.929 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:38.925+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:39.190 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:39.183+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:39.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:39.290+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:39.330 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:39.316+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:39.350 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:39.344+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:39.352 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:39.347+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:39.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:39.376+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:39.754 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:39.748+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:39.976 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:26:39.971+0000 1694f700 -1 osd.0 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.578626+0000 front 2019-12-11T22:25:11.575605+0000 (oldest deadline 2019-12-11T22:25:54.317800+0000) 2019-12-11T22:26:40.175 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:26:40.168+0000 1694f700 -1 osd.4 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:07.179772+0000 front 2019-12-11T22:25:07.186151+0000 (oldest deadline 2019-12-11T22:25:52.447276+0000) 2019-12-11T22:26:40.292 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:26:40.287+0000 1694f700 -1 osd.2 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:12.051015+0000 front 2019-12-11T22:25:12.049540+0000 (oldest deadline 2019-12-11T22:25:54.664027+0000) 2019-12-11T22:26:40.318 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:26:40.304+0000 1694f700 -1 osd.6 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:11.443508+0000 front 2019-12-11T22:25:11.442927+0000 (oldest deadline 2019-12-11T22:25:56.679311+0000) 2019-12-11T22:26:40.378 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:40.371+0000 1694f700 -1 osd.3 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.253512+0000 front 2019-12-11T22:25:09.276648+0000 (oldest deadline 2019-12-11T22:25:55.083242+0000) 2019-12-11T22:26:40.380 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:40.376+0000 1694f700 -1 osd.3 289 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:40.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:26:40.419+0000 1694f700 -1 osd.7 289 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:25:09.119428+0000 front 2019-12-11T22:25:09.174675+0000 (oldest deadline 2019-12-11T22:25:52.430237+0000) 2019-12-11T22:26:40.722 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:26:40.712+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 47sec e289 v289) 2019-12-11T22:26:40.749 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:40.743+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e287) 2019-12-11T22:26:41.221 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:26:41.131+0000 34f8d700 -1 osd.5 290 build_incremental_map_msg missing incremental map 289 2019-12-11T22:26:41.680 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:26:41.670+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:109515 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 442599088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e285) 2019-12-11T22:26:43.320 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:26:43.294+0000 20e25700 -1 RGWWatcher::handle_error cookie 442599088 err (107) Transport endpoint is not connected 2019-12-11T22:26:46.065 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:54:32.996 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.089 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:54:33.020 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.154 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:54:33.085 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.158 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:54:33.089 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.191 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:54:33.093 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.197 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:54:33.099 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.256 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:54:33.187 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.287 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:54:33.217 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:26:46.371 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:54:33.269 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:27:07.541 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:27:07.519+0000 31786700 -1 osd.3 293 build_incremental_map_msg missing incremental map 293 2019-12-11T22:27:07.595 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:27:07.536+0000 31786700 -1 osd.3 293 build_incremental_map_msg missing incremental map 293 2019-12-11T22:27:07.607 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:27:07.539+0000 31786700 -1 osd.3 293 build_incremental_map_msg missing incremental map 293 2019-12-11T22:27:07.856 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:27:07.590+0000 3378a700 -1 osd.3 293 build_incremental_map_msg missing incremental map 293 2019-12-11T22:27:07.927 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:27:07.748+0000 31f87700 -1 osd.3 293 build_incremental_map_msg missing incremental map 293 2019-12-11T22:28:08.537 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:08.528+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:08.680 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:08.675+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:09.511 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:09.505+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:09.586 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:09.581+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:09.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:09.691+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:09.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:09.966+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:10.461 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:10.455+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:10.551 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:10.546+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:10.562 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:10.557+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:10.704 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:10.699+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:10.865 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:10.858+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:10.963 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:10.958+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:11.497 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:11.493+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:11.554 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:11.550+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:11.583 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:11.575+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:11.612 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:11.607+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:11.719 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:11.712+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:11.869 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:11.864+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:11.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:11.928+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:12.472 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:12.467+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:12.549 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:12.542+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:12.558 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:12.552+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:12.628 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:12.623+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:12.736 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:12.727+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:12.864 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:12.859+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:12.953 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:12.947+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:13.464 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:13.460+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:13.521 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:13.517+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:13.542 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:13.534+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:13.653 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:13.648+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:13.714 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:13.708+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:13.824 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:13.819+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:13.969 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:13.962+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:14.457 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:14.448+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:14.558 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:14.552+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:14.592 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:14.586+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:14.663 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:14.659+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:14.707 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:14.700+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:14.795 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:14.790+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:14.968 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:14.961+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:15.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:15.464+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:15.600 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:15.595+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:15.623 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:15.617+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:15.684 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:15.679+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:15.744 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:15.735+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:15.749 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:15.745+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:16.003 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:15.997+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:16.464 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:16.458+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:16.565 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:16.560+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:16.641 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:16.634+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:16.728 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:16.721+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:16.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:16.731+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:16.769 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:16.765+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:17.023 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:17.016+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:17.490 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:17.485+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:17.610 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:17.605+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:17.668 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:17.661+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:17.758 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:17.751+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:17.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:17.753+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:17.761 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:17.757+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:18.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:18.063+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:18.464 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:18.460+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:18.594 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:18.588+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:18.706 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:18.699+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:18.741 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:18.737+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:18.794 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:18.788+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:18.815 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:18.810+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:19.124 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:19.117+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:19.477 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:19.472+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:19.570 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:19.563+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:19.677 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:19.669+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:19.762 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:19.757+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:19.793 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:19.787+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:19.797 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:19.791+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:20.146 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:20.139+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:20.484 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:20.479+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:20.572 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:20.566+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:20.660 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:20.653+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:20.758 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:20.751+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:20.790 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:20.785+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:20.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:20.805+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:21.170 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:21.163+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:21.533 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:21.528+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:21.568 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:21.563+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:21.694 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:21.687+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:21.756 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:21.749+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:21.796 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:21.791+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:21.799 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:21.794+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:22.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:22.213+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:22.567 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:22.553+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:22.616 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:22.611+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:22.699 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:22.688+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:22.770 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:22.762+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:22.791 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:22.787+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:22.827 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:22.823+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:23.174 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:23.168+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:23.605 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:23.600+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:23.608 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:23.604+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:23.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:23.719+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:23.785 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:23.777+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:23.827 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:23.821+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:23.827 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:23.821+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:24.208 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:24.202+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:24.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:24.570+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:24.629 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:24.623+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:24.693 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:24.685+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:24.796 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:24.789+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:24.797 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:24.792+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:24.829 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:24.824+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:25.253 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:25.247+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:25.529 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:25.524+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:25.659 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:25.655+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:25.713 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:25.707+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:25.752 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:25.746+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:25.844 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:25.838+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:25.881 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:25.876+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:26.267 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:26.259+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:26.486 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:26.482+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:26.692 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:26.685+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:26.711 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:26.707+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:26.792 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:26.787+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:26.820 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:26.813+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:26.852 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:26.847+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:27.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:27.272+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:27.519 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:27.514+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:27.744 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:27.737+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:27.758 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:27.754+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:27.804 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:27.800+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:27.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:27.805+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:27.837 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:27.830+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:28.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:28.289+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:28.539 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:28.534+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:28.713 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:28.708+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:28.764 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:28.759+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:28.765 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:28.758+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:28.820 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:28.813+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:28.835 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:28.830+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:29.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:29.246+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:29.505 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:29.497+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:29.694 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:29.689+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:29.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:29.731+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:29.810 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:29.805+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:29.833 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:29.828+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:29.846 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:29.841+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:30.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:30.265+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:30.495 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:30.490+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:30.731 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:30.726+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:30.764 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:30.758+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:30.765 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:30.760+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:30.803 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:30.798+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:30.828 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:30.823+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:31.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:31.304+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:31.499 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:31.493+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:31.704 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:31.699+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:31.803 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:31.798+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:31.809 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:31.803+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:31.824 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:31.817+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:31.863 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:31.859+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:32.361 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:32.355+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:32.477 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:32.472+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:32.732 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:32.727+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:32.790 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:32.785+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:32.809 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:32.803+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:32.829 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:32.818+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:32.906 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:32.899+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:33.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:33.321+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:33.442 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:33.436+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:33.729 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:33.724+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:33.813 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:33.806+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:33.818 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:33.812+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:33.863 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:33.856+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:33.894 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:33.888+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:34.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:34.279+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:34.410 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:34.405+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:34.772 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:34.767+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:34.829 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:34.821+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:34.839 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:34.827+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:34.840 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:34.834+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:34.945 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:34.939+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:35.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:35.329+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:35.370 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:35.364+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:35.768 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:35.764+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:35.801 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:35.795+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:35.875 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:35.868+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:35.877 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:35.871+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:35.922 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:35.917+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:36.383 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:36.376+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:36.402 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:36.397+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:36.736 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:36.732+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:36.811 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:36.805+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:36.853 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:36.847+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:36.856 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:36.849+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:36.895 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:36.890+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:37.388 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:37.383+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:37.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:37.422+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:37.773 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:37.761+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:37.794 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:37.789+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:37.835 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:37.829+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:37.849 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:37.841+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:37.868 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:37.863+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:38.391 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:38.384+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:38.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:38.464+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:38.762 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:38.755+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:38.814 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:38.807+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:38.833 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:38.827+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:38.845 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:38.838+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:38.893 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:38.886+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:39.404 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:39.397+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:39.503 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:39.496+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:39.783 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:39.776+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:39.784 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:39.778+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:39.832 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:39.826+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:39.847 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:39.841+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:39.917 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:39.911+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:40.392 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:40.386+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:40.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:40.514+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:40.801 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:40.789+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:40.824 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:40.818+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:40.872 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:40.866+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:40.886 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:40.880+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:40.906 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:40.899+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:41.330 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:28:41.323+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 44sec e294 v294) 2019-12-11T22:28:41.441 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:41.435+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:41.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:41.567+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:41.827 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:41.819+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:41.838 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:41.832+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:41.856 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:41.850+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:41.874 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:41.868+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:41.880 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:41.875+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:42.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:42.455+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:42.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:42.557+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:42.847 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:42.839+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:42.862 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:42.855+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:42.878 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:42.872+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:42.879 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:42.873+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:42.901 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:42.895+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:43.424 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:43.418+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:43.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:43.597+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:43.877 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:43.868+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:43.908 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:43.901+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:43.913 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:43.907+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:43.913 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:43.906+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:43.931 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:43.925+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:44.418 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:44.411+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:44.626 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:44.619+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:44.910 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:44.904+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:44.923 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:44.919+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:44.937 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:44.931+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:44.937 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:44.932+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:44.951 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:44.944+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:45.379 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:45.374+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:45.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:45.619+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:45.884 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:28:45.879+0000 1694f700 -1 osd.0 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719404+0000 front 2019-12-11T22:27:25.975623+0000 (oldest deadline 2019-12-11T22:28:10.670394+0000) 2019-12-11T22:28:45.902 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:28:45.897+0000 1694f700 -1 osd.1 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.623010+0000 front 2019-12-11T22:27:25.664163+0000 (oldest deadline 2019-12-11T22:28:09.100275+0000) 2019-12-11T22:28:45.905 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:28:45.899+0000 1694f700 -1 osd.2 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.826111+0000 front 2019-12-11T22:27:26.874640+0000 (oldest deadline 2019-12-11T22:28:10.111447+0000) 2019-12-11T22:28:45.932 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:28:45.925+0000 1694f700 -1 osd.4 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:26.057712+0000 front 2019-12-11T22:27:26.004370+0000 (oldest deadline 2019-12-11T22:28:07.944113+0000) 2019-12-11T22:28:45.962 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:28:45.956+0000 1694f700 -1 osd.6 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:27.342992+0000 front 2019-12-11T22:27:27.350954+0000 (oldest deadline 2019-12-11T22:28:08.188617+0000) 2019-12-11T22:28:46.400 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:28:46.396+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 44sec e294 v294) 2019-12-11T22:28:46.425 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:28:46.419+0000 1694f700 -1 osd.3 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:24.991217+0000 front 2019-12-11T22:27:25.199374+0000 (oldest deadline 2019-12-11T22:28:09.642948+0000) 2019-12-11T22:28:46.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:28:46.643+0000 1694f700 -1 osd.7 294 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:27:25.719868+0000 front 2019-12-11T22:27:25.461050+0000 (oldest deadline 2019-12-11T22:28:09.449575+0000) 2019-12-11T22:28:59.243 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:56:46.174 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.255 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:56:46.185 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.255 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:56:46.185 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.260 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:56:46.191 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.267 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:56:46.167 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.273 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:56:46.173 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.378 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:05:56:46.309 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.411 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:05:56:46.312 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:28:59.446 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:05:56:46.376 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:29:16.741 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:16.736+0000 1694f700 -1 osd.4 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109849 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 67] snapc 0=[] ondisk+write+known_if_redirected e295) 2019-12-11T22:29:17.757 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:17.752+0000 1694f700 -1 osd.4 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109849 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 67] snapc 0=[] ondisk+write+known_if_redirected e295) 2019-12-11T22:29:18.801 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:18.796+0000 1694f700 -1 osd.4 296 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109849 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 67] snapc 0=[] ondisk+write+known_if_redirected e295) 2019-12-11T22:29:21.191 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:21.166+0000 35f8f700 -1 osd.0 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.192 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:21.171+0000 3578e700 -1 osd.0 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.200 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:21.179+0000 35f8f700 -1 osd.0 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:21.262+0000 35b8e700 -1 osd.7 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.396 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:21.266+0000 35b8e700 -1 osd.7 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.525 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:21.378+0000 35f8f700 -1 osd.0 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.634 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:21.406+0000 35b8e700 -1 osd.7 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:21.448+0000 35b8e700 -1 osd.7 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.691 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:21.522+0000 32b88700 -1 osd.7 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.776 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:21.569+0000 35b8e700 -1 osd.7 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.781 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:21.565+0000 37f93700 -1 osd.2 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.785 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:21.668+0000 38794700 -1 osd.0 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.787 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:21.586+0000 38794700 -1 osd.2 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.861 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:29:21.730+0000 36f91700 -1 osd.1 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.861 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:21.691+0000 3478c700 -1 osd.2 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.873 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:21.728+0000 3638f700 -1 osd.7 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:21.933 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:21.926+0000 35f8f700 -1 osd.2 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:22.030 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:29:21.862+0000 36f91700 -1 osd.1 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:22.241 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:29:21.974+0000 3378a700 -1 osd.1 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:22.241 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:29:21.976+0000 3378a700 -1 osd.1 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:22.293 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:29:22.115+0000 3378a700 -1 osd.1 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:22.391 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:29:22.309+0000 31786700 -1 osd.1 298 build_incremental_map_msg missing incremental map 298 2019-12-11T22:29:43.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:43.908+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:44.568 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:44.559+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:44.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:44.903+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:44.980 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:44.975+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:45.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:45.528+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:45.943 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:45.938+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:45.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:45.969+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:45.983 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:45.977+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:46.585 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:46.580+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:46.939 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:46.933+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:47.007 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:47.002+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:47.014 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:47.006+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:47.442 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:47.433+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:47.592 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:47.586+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:47.894 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:47.889+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:47.966 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:47.960+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:48.032 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:48.025+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:48.484 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:48.469+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:48.585 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:48.580+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:48.929 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:48.925+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:48.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:48.925+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:48.932 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:48.928+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:48.966 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:48.958+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:49.016 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:49.009+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:49.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:49.447+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:49.570 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:49.566+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:49.907 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:49.901+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:49.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:49.904+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:49.918 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:49.909+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:49.970 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:49.962+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:50.020 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:50.012+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:50.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:50.478+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:50.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:50.575+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:50.873 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:50.866+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:50.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:50.868+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:50.930 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:50.922+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:50.963 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:50.955+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:51.049 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:51.042+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:51.490 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:51.482+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:51.548 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:51.543+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:51.909 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:51.904+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:51.909 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:51.905+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:51.916 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:51.909+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:51.961 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:51.953+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:52.019 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:52.011+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:52.512 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:52.508+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:52.531 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:52.521+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:52.910 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:52.903+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:52.921 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:52.910+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:52.921 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:52.912+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:52.935 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:29:52.929+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:29:52.978 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:52.971+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:52.992 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:52.986+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:53.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:53.496+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:53.557 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:53.553+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:53.883 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:53.875+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:53.940 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:29:53.926+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:29:53.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:53.935+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:53.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:53.936+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:53.968 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:53.958+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:54.037 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:54.033+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:54.557 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:54.550+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:54.592 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:54.587+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:54.900 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:54.894+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:54.938 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:54.933+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:54.939 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:29:54.932+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:29:54.939 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:54.934+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:54.986 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:54.981+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:55.024 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:55.019+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:55.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:55.512+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:55.564 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:55.560+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:55.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:55.907+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:55.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:55.910+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:55.915 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:55.908+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:55.938 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:29:55.933+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:29:55.974 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:29:55.968+0000 1694f700 -1 osd.4 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.968458+0000 front 2019-12-11T22:29:00.969845+0000 (oldest deadline 2019-12-11T22:29:45.632999+0000) 2019-12-11T22:29:56.018 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:56.014+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:56.561 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:56.554+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:56.613 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:56.609+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:56.921 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:29:56.915+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:29:56.924 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:56.921+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:56.926 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:56.922+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:56.931 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:56.925+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:57.015 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:57.010+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:57.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:57.525+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:57.624 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:57.620+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:57.888 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:57.882+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:57.888 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:57.883+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:57.960 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:57.952+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:57.968 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:29:57.962+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:29:58.040 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:58.034+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:58.535 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:58.527+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:58.628 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:58.617+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:58.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:58.901+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:58.916 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:58.906+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:58.958 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:58.950+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:58.980 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:29:58.974+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:29:59.012 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:59.006+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:29:59.565 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:29:59.558+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:29:59.625 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:29:59.619+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:29:59.876 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:59.869+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:29:59.877 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:29:59.872+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:29:59.924 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:29:59.918+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:29:59.983 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:29:59.979+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:00.025 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:00.019+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:00.028 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:00.024+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:00.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:00.547+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:00.660 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:00.654+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:00.864 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:00.859+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:00.866 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:00.863+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:00.953 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:00.946+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:01.032 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:01.027+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:01.040 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:01.032+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:01.043 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:01.036+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:01.611 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:01.602+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:01.687 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:01.683+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:01.900 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:01.894+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:01.901 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:01.896+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:01.930 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:01.923+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:02.047 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:02.041+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:02.092 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:02.086+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:02.101 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:02.092+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:02.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:02.599+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:02.738 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:02.733+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:02.936 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:02.932+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:02.939 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:02.935+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:02.962 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:02.955+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:03.069 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:03.063+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:03.075 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:03.068+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:03.085 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:03.077+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:03.654 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:03.647+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:03.731 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:03.725+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:03.912 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:03.906+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:03.914 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:03.911+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:03.925 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:03.919+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:04.082 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:04.077+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:04.113 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:04.105+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:04.114 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:04.108+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:04.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:04.639+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:04.777 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:04.772+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:04.879 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:04.874+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:04.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:04.879+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:04.980 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:04.973+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:05.090 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:05.083+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:05.091 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:05.086+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:05.118 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:05.114+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:05.621 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:05.615+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:05.782 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:05.777+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:05.917 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:05.911+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:05.917 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:05.913+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:06.020 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:06.013+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:06.100 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:06.092+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:06.100 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:06.095+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:06.158 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:06.152+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:06.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:06.585+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:06.824 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:06.816+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:06.880 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:06.872+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:06.881 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:06.874+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:07.007 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:07.000+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:07.118 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:07.111+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:07.119 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:07.114+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:07.181 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:07.172+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:07.567 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:07.559+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:07.856 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:07.851+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:07.872 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:07.865+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:07.872 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:07.866+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:08.025 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:08.018+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:08.129 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:08.122+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:08.133 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:08.127+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:08.196 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:08.190+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:08.548 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:08.541+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:08.846 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:08.840+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:08.869 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:08.863+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:08.870 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:08.864+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:09.005 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:08.997+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:09.186 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:09.177+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:09.187 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:09.181+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:09.249 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:09.243+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:09.509 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:09.502+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:09.809 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:09.802+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:09.860 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:09.853+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:09.860 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:09.855+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:10.051 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:10.044+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:10.175 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:10.168+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:10.179 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:10.173+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:10.233 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:10.226+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:10.557 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:10.550+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:10.839 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:10.832+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:10.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:10.896+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:10.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:10.897+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:11.041 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:11.035+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:11.154 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:11.148+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:11.157 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:11.152+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:11.283 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:11.272+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:11.538 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:11.531+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:11.849 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:11.843+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:11.879 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:11.873+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:11.881 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:11.875+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:12.066 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:12.059+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:12.136 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:12.123+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:12.137 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:12.129+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:12.323 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:12.316+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:12.538 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:12.530+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:12.836 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:12.829+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:12.838 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:12.833+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:12.881 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:12.872+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:13.094 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:13.086+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:13.095 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:13.089+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:13.114 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:13.106+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:13.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:13.324+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:13.557 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:13.549+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:13.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:13.822+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:13.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:13.823+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:13.881 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:13.874+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:14.093 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:14.086+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:14.104 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:14.097+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:14.163 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:14.155+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:14.332 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:14.324+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:14.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:14.565+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:14.834 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:14.827+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:14.834 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:14.830+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:14.878 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:14.873+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:15.081 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:15.074+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:15.085 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:15.079+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:15.161 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:15.153+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:15.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:15.362+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:15.557 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:15.549+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:15.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:15.837+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:15.854 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:15.842+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:15.926 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:15.921+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:16.059 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:16.047+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:16.059 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:16.050+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:16.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:16.163+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:16.351 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:16.345+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:16.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:16.595+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:16.653 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:30:16.649+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e300 v300) 2019-12-11T22:30:16.824 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:16.819+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:16.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:16.823+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:16.936 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:16.932+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:17.035 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:17.027+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:17.037 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:17.032+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:17.199 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:17.191+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:17.344 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:17.334+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:17.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:17.618+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:17.799 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:17.793+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:17.800 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:17.795+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:17.975 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:17.970+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:18.077 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:18.070+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:18.080 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:18.075+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:18.187 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:18.179+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:18.330 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:18.325+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:18.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:18.598+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:18.823 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:18.818+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:18.826 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:18.821+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:18.999 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:18.994+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:19.090 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:19.082+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:19.091 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:19.085+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:19.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:19.174+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:19.319 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:19.315+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:19.569 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:19.561+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:19.854 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:19.847+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:19.855 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:19.851+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:19.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:19.949+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:20.124 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:20.116+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:20.125 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:20.119+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:20.144 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:20.136+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:20.322 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:20.317+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:20.586 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:20.578+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:20.866 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:20.861+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:20.870 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:20.866+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:20.924 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:20.919+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:21.114 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:21.105+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:21.115 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:21.108+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:21.122 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:21.116+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:21.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:21.331+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:21.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:21.621+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:21.680 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:30:21.668+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e300 v300) 2019-12-11T22:30:21.874 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:21.869+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:21.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:21.871+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:21.917 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:21.912+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:22.091 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:22.085+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:22.095 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:22.089+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:22.133 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:22.127+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:22.347 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:22.343+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:22.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:22.593+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:22.882 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:22.876+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:22.882 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:22.878+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:22.897 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:22.892+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:23.094 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:23.086+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:23.095 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:23.089+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:23.189 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:23.181+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:23.385 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:23.380+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:23.590 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:23.583+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:23.860 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:23.854+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:23.874 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:23.869+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:23.882 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:23.872+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:24.144 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:24.135+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:24.145 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:24.138+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:24.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:24.203+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:24.354 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:24.350+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:24.642 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:24.636+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:24.844 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:24.838+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:24.845 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:24.841+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:24.906 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:24.901+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:25.103 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:25.095+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:25.104 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:25.097+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:25.208 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:25.200+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:25.372 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:25.367+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:25.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:25.618+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:25.839 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:25.832+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:25.839 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:25.836+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:25.959 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:25.954+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:26.060 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:26.051+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:26.061 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:26.054+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:26.258 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:26.251+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:26.342 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:26.336+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:26.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:26.596+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:26.683 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:30:26.679+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e300 v300) 2019-12-11T22:30:26.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:26.822+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:26.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:26.823+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:26.931 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:26.927+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:27.028 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:27.023+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:27.030 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:27.025+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:27.246 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:27.240+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:27.373 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:27.366+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:27.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:27.637+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:27.859 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:27.846+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:27.860 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:27.848+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:27.950 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:27.946+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:27.990 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:27.983+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:27.990 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:27.985+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:28.289 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:28.283+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:28.404 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:28.400+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:28.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:28.616+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:28.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:28.837+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:28.843 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:28.839+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:28.969 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:28.964+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:28.997 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:28.988+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:28.997 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:28.992+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:29.305 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:29.297+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:29.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:29.402+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:29.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:29.580+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:29.838 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:29.833+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:29.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:29.836+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:29.999 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:29.993+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:30.014 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:30.004+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:30.016 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:30.007+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:30.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:30.289+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:30.380 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:30.374+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:30.614 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:30.607+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:30.816 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:30.811+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:30.819 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:30.815+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:30.954 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:30:30.948+0000 1694f700 -1 osd.2 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:01.562897+0000 front 2019-12-11T22:29:01.563811+0000 (oldest deadline 2019-12-11T22:29:44.422347+0000) 2019-12-11T22:30:30.969 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:30.961+0000 1694f700 -1 osd.5 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:29:19.338255+0000 (oldest deadline 2019-12-11T22:29:59.338255+0000) 2019-12-11T22:30:30.970 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:30:30.964+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e298) 2019-12-11T22:30:31.327 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:30:31.320+0000 1694f700 -1 osd.6 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.708300+0000 front 2019-12-11T22:29:03.709320+0000 (oldest deadline 2019-12-11T22:29:48.646159+0000) 2019-12-11T22:30:31.427 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:30:31.421+0000 1694f700 -1 osd.0 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:00.869764+0000 front 2019-12-11T22:29:00.826522+0000 (oldest deadline 2019-12-11T22:29:44.243590+0000) 2019-12-11T22:30:31.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:30:31.567+0000 1694f700 -1 osd.7 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:02.189117+0000 front 2019-12-11T22:29:02.497678+0000 (oldest deadline 2019-12-11T22:29:46.851680+0000) 2019-12-11T22:30:31.742 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:30:31.731+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 42sec e300 v300) 2019-12-11T22:30:31.812 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:31.807+0000 1694f700 -1 osd.3 300 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:29:03.785262+0000 front 2019-12-11T22:29:00.885263+0000 (oldest deadline 2019-12-11T22:29:43.728760+0000) 2019-12-11T22:30:31.814 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:31.808+0000 1694f700 -1 osd.3 300 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:32.859 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:30:32.856+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:109908 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5482781872 gen 1] snapc 0=[] ondisk+write+known_if_redirected e297) 2019-12-11T22:30:34.051 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:30:33.981+0000 20e25700 -1 RGWWatcher::handle_error cookie 5482781872 err (107) Transport endpoint is not connected 2019-12-11T22:32:01.873 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:01.867+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:02.262 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:02.257+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:02.892 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:02.888+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:03.262 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:03.255+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:03.839 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:03.834+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:03.881 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:03.877+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:03.956 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:03.951+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:04.277 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:04.271+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:04.799 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:04.794+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:04.932 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:04.928+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:04.991 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:04.987+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:05.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:05.289+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:05.493 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:05.486+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:05.776 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:05.771+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:05.948 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:05.943+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:06.033 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:06.026+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:06.269 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:06.262+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:06.535 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:06.529+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:06.757 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:06.751+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:06.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:06.938+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:07.014 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:07.008+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:07.301 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:07.295+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:07.514 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:07.506+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:07.747 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:07.737+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:07.921 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:07.916+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:08.031 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:08.026+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:08.311 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:08.302+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:08.537 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:08.530+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:08.715 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:08.708+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:08.892 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:08.887+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:08.983 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:08.979+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:09.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:09.308+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:09.573 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:09.566+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:09.705 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:09.701+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:09.859 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:09.854+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:09.986 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:09.979+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:10.349 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:10.340+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:10.591 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:10.585+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:10.700 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:10.695+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:10.821 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:10.816+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:10.946 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:10.941+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:11.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:11.366+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:11.631 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:11.624+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:11.727 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:11.723+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:11.838 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:11.833+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:11.994 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:11.990+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:12.351 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:12.345+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:12.668 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:12.662+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:12.764 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:12.758+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:12.858 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:12.852+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:13.022 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:13.017+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:13.360 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:13.354+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:13.719 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:13.713+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:13.745 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:13.740+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:13.847 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:13.842+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:14.026 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:14.020+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:14.355 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:14.348+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:14.717 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:14.711+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:14.735 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:14.729+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:14.740 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:14.734+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:14.886 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:14.881+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:15.040 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:15.034+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:15.404 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:15.397+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:15.693 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:15.686+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:15.716 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:15.710+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:15.754 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:15.749+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:15.909 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:15.905+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:16.054 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:16.049+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:16.292 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:16.287+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:16.401 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:16.395+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:16.707 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:16.700+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:16.772 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:16.767+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:16.775 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:16.758+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:16.919 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:16.914+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:17.062 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:17.057+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:17.261 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:17.256+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:17.399 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:17.392+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:17.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:17.682+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:17.729 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:17.722+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:17.817 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:17.812+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:17.876 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:17.871+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:18.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:18.084+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:18.250 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:18.243+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:18.368 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:18.361+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:18.711 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:18.701+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:18.720 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:18.714+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:18.781 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:18.775+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:18.911 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:18.905+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:19.042 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:19.037+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:19.288 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:19.283+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:19.349 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:19.342+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:19.716 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:19.708+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:19.723 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:19.717+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:19.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:19.772+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:19.879 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:19.872+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:20.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:20.082+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:20.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:20.309+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:20.332 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:20.326+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:20.693 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:20.686+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:20.746 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:20.740+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:20.759 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:20.752+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:20.877 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:20.870+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:21.073 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:21.067+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:21.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:21.323+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:21.358 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:21.352+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:21.731 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:21.724+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:21.745 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:21.739+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:21.749 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:21.744+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:21.909 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:21.902+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:22.051 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:22.045+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:22.331 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:22.325+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:22.381 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:22.377+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:22.712 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:22.706+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:22.783 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:22.776+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:22.795 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:22.788+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:22.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:22.949+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:23.091 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:23.085+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:23.334 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:23.326+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:23.429 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:23.422+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:23.685 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:23.679+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:23.770 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:23.764+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:23.831 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:23.825+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:23.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:23.978+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:24.127 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:24.121+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:24.288 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:24.281+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:24.410 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:24.404+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:24.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:24.647+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:24.761 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:24.755+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:24.877 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:24.871+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:24.973 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:24.964+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:25.121 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:25.116+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:25.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:25.322+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:25.390 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:25.384+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:25.640 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:25.634+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:25.786 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:25.780+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:25.917 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:25.910+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:25.986 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:25.981+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:26.147 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:26.143+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:26.321 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:26.314+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:26.443 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:26.436+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:26.599 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:26.593+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:26.777 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:26.771+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:26.929 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:26.922+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:26.984 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:26.979+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:27.107 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:27.103+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:27.357 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:27.350+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:27.459 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:27.454+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:27.645 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:27.639+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:27.784 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:27.773+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:27.940 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:27.935+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:27.947 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:27.940+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:28.147 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:28.142+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:28.318 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:28.311+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:28.425 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:28.420+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:28.636 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:28.629+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:28.742 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:28.737+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:28.926 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:28.918+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:28.949 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:28.943+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:29.124 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:29.117+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:29.304 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:29.293+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:29.409 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:29.401+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:29.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:29.631+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:29.697 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:29.692+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:29.925 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:29.919+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:29.939 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:29.934+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:30.135 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:30.129+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:30.277 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:30.271+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:30.378 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:30.373+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:30.671 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:30.664+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:30.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:30.729+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:30.966 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:30.960+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:30.979 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:30.972+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:31.096 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:31.091+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:31.240 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:31.233+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:31.397 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:31.392+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:31.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:31.624+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:31.701 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:31.696+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:31.967 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:31.960+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:31.997 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:31.993+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:32.099 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:32.093+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:32.145 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:32:32.141+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 38sec e305 v305) 2019-12-11T22:32:32.208 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:32.203+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:32.404 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:32.398+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:32.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:32.597+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:32.748 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:32.743+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:32.978 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:32.974+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:33.016 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:33.011+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:33.145 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:33.141+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:33.201 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:33.194+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:33.363 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:33.358+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:33.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:33.620+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:33.784 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:33.780+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:33.981 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:33.974+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:34.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:34.030+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:34.181 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:34.175+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:34.199 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:34.194+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:34.367 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:34.362+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:34.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:34.662+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:34.787 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:34.781+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:35.007 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:34.995+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:35.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:35.010+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:35.136 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:35.130+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:35.199 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:35.195+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:35.362 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:35.358+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:35.672 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:35.665+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:35.810 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:35.805+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:35.959 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:35.952+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:35.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:35.980+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:36.190 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:36.183+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:36.204 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:36.199+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:36.356 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:36.349+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:36.663 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:36.657+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:36.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:36.808+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:36.942 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:36.934+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:37.040 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:37.034+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:37.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:37.175+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:37.197 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:32:37.192+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 38sec e305 v305) 2019-12-11T22:32:37.206 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:37.199+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:37.348 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:37.344+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:37.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:37.662+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:37.831 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:37.827+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:37.988 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:37.981+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:38.082 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:38.078+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:38.228 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:38.222+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:38.242 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:38.237+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:38.403 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:38.396+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:38.638 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:38.633+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:38.787 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:38.782+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:39.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:39.032+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:39.101 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:39.097+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:39.200 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:39.188+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:39.237 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:39.226+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:39.449 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:39.443+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:39.594 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:39.589+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:39.829 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:39.824+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:40.004 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:39.998+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:40.069 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:40.064+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:40.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:40.189+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:40.225 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:40.220+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:40.499 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:40.494+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:40.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:40.553+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:40.822 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:40.817+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:40.994 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:40.987+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:41.022 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:41.017+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:41.187 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:41.181+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:41.274 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:41.269+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:41.456 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:32:41.452+0000 1694f700 -1 osd.3 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:21.673997+0000 front 2019-12-11T22:31:20.981938+0000 (oldest deadline 2019-12-11T22:32:15.814495+0000) 2019-12-11T22:32:41.583 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:32:41.576+0000 1694f700 -1 osd.7 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:18.636198+0000 front 2019-12-11T22:31:17.102778+0000 (oldest deadline 2019-12-11T22:32:14.498557+0000) 2019-12-11T22:32:41.784 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:32:41.779+0000 1694f700 -1 osd.0 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:20.410036+0000 front 2019-12-11T22:31:19.971710+0000 (oldest deadline 2019-12-11T22:32:02.843253+0000) 2019-12-11T22:32:42.020 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:32:42.014+0000 1694f700 -1 osd.4 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:19.758778+0000 front 2019-12-11T22:31:23.086285+0000 (oldest deadline 2019-12-11T22:32:05.037662+0000) 2019-12-11T22:32:42.028 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:32:42.022+0000 1694f700 -1 osd.2 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.601630+0000 front 2019-12-11T22:31:23.616064+0000 (oldest deadline 2019-12-11T22:32:01.365456+0000) 2019-12-11T22:32:42.214 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:32:42.207+0000 1694f700 -1 osd.6 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:22.030704+0000 front 2019-12-11T22:31:22.028520+0000 (oldest deadline 2019-12-11T22:32:02.079037+0000) 2019-12-11T22:32:42.274 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:32:42.265+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 38sec e305 v305) 2019-12-11T22:32:42.320 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:32:42.315+0000 1694f700 -1 osd.1 305 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:31:23.604636+0000 front 2019-12-11T22:31:23.087858+0000 (oldest deadline 2019-12-11T22:32:03.426976+0000) 2019-12-11T22:32:54.559 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:00:41.490 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.566 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:00:41.497 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.573 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:00:41.503 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.576 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:00:41.507 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.577 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:00:41.478 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.580 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:00:41.482 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.804 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:00:41.735 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.827 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:00:41.728 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:32:54.843 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:00:41.773 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.451 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:00:49.382 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.455 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:00:49.387 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.479 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:00:49.406 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.505 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:00:49.401 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.509 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:00:49.410 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.509 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:00:49.437 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.510 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:00:49.439 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.525 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:00:49.456 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:02.538 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:00:49.439 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:33:15.737 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:15.712+0000 31f87700 -1 osd.0 309 build_incremental_map_msg missing incremental map 309 2019-12-11T22:33:15.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:15.753+0000 3578e700 -1 osd.0 309 build_incremental_map_msg missing incremental map 309 2019-12-11T22:33:15.887 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:15.866+0000 31f87700 -1 osd.0 309 build_incremental_map_msg missing incremental map 309 2019-12-11T22:33:16.167 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:15.982+0000 33f8b700 -1 osd.0 309 build_incremental_map_msg missing incremental map 309 2019-12-11T22:33:16.409 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:16.255+0000 3478c700 -1 osd.0 309 build_incremental_map_msg missing incremental map 309 2019-12-11T22:33:40.440 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:40.432+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:41.028 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:41.022+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:41.392 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:41.386+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:41.527 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:41.519+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:41.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:41.700+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:42.077 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:42.067+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:42.356 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:42.350+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:42.573 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:42.565+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:42.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:42.569+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:42.714 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:42.706+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:43.090 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:43.085+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:43.342 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:43.337+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:43.576 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:43.567+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:43.626 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:43.620+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:43.746 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:43.737+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:43.915 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:43.904+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:44.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:44.096+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:44.367 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:44.363+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:44.628 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:44.622+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:44.646 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:44.642+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:44.736 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:44.724+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:44.902 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:44.896+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:45.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:45.115+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:45.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:45.117+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:45.337 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:45.328+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:45.644 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:45.634+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:45.653 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:45.646+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:45.707 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:45.699+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:45.919 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:45.910+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:46.137 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:46.130+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:46.137 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:46.132+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:46.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:46.287+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:46.644 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:46.637+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:46.647 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:46.643+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:46.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:46.720+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:46.961 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:46.953+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:47.174 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:47.169+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:47.175 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:47.171+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:47.259 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:47.251+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:47.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:47.634+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:47.647 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:47.639+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:47.688 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:47.678+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:48.019 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:48.007+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:48.062 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:48.055+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:48.212 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:48.206+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:48.212 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:48.207+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:48.273 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:48.266+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:48.651 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:48.644+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:48.678 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:48.672+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:48.704 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:48.697+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:49.006 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:48.988+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:49.031 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:49.024+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:49.197 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:49.190+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:49.197 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:49.192+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:49.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:49.271+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:49.674 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:49.668+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:49.689 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:49.683+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:49.706 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:49.700+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:49.962 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:49.955+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:50.028 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:50.022+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:50.162 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:50.153+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:50.162 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:50.155+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:50.280 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:50.274+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:50.628 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:50.622+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:50.680 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:50.673+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:50.695 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:50.689+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:50.997 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:50.989+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:51.026 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:51.017+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:51.186 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:51.180+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:51.187 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:51.181+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:51.251 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:51.244+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:51.612 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:51.605+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:51.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:51.700+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:51.715 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:51.708+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:52.033 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:52.026+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:52.055 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:52.049+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:52.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:52.227+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:52.296 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:52.228+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:52.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:52.257+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:52.593 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:52.586+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:52.665 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:52.660+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:52.689 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:52.682+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:53.072 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:53.063+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:53.084 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:53.079+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:53.268 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:53.262+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:53.270 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:53.265+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:53.305 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:53.296+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:53.552 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:53.545+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:53.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:53.618+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:53.693 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:53.682+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:54.096 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:54.092+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:54.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:54.105+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:54.302 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:54.296+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:54.302 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:54.298+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:54.336 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:54.332+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:54.522 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:54.518+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:54.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:54.629+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:54.652 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:54.645+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:55.065 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:55.058+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:33:55.067 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:55.063+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:55.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:55.093+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:55.307 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:55.300+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:55.307 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:55.302+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:55.362 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:55.356+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:55.542 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:55.536+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:55.638 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:55.631+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:55.640 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:55.636+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:56.044 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:56.039+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:33:56.047 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:56.042+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:56.156 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:56.149+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:56.317 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:56.312+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:56.317 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:56.313+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:56.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:56.347+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:56.580 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:56.575+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:56.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:56.653+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:56.690 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:56.683+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:57.094 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:57.087+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:33:57.095 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:57.090+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:57.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:57.200+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:57.338 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:57.332+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:57.339 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:57.334+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:57.376 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:57.372+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:57.540 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:57.533+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:57.629 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:57.622+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:57.661 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:57.654+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:58.074 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:58.067+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:33:58.075 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:58.070+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:58.199 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:58.192+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:58.388 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:58.378+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:58.393 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:58.382+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:58.418 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:58.412+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:58.575 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:58.570+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:58.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:58.603+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:58.678 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:58.672+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:33:59.044 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:59.037+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:33:59.045 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:33:59.040+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:33:59.225 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:33:59.219+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:33:59.353 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:59.348+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:33:59.354 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:33:59.350+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:33:59.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:33:59.456+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:33:59.563 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:33:59.558+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:33:59.646 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:33:59.639+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:33:59.654 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:33:59.647+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:00.044 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:00.037+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:00.045 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:00.039+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:00.269 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:00.255+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:00.312 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:00.308+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:00.315 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:00.310+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:00.416 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:00.411+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:00.588 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:00.579+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:00.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:00.614+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:00.694 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:00.683+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:01.024 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:01.017+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:01.028 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:01.022+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:01.281 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:01.274+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:01.316 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:01.311+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:01.316 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:01.312+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:01.414 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:01.409+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:01.582 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:01.573+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:01.622 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:01.614+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:01.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:01.714+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:02.049 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:02.041+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:02.050 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:02.044+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:02.247 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:02.238+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:02.295 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:02.289+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:02.296 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:02.291+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:02.368 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:02.362+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:02.538 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:02.533+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:02.593 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:02.585+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:02.767 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:02.760+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:03.046 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:03.039+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:03.047 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:03.041+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:03.214 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:03.205+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:03.325 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:03.320+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:03.326 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:03.322+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:03.391 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:03.387+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:03.578 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:03.572+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:03.628 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:03.621+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:03.781 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:03.773+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:04.036 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:04.028+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:04.048 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:04.033+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:04.262 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:04.255+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:04.299 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:04.292+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:04.300 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:04.294+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:04.412 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:04.406+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:04.619 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:04.614+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:04.654 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:04.647+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:04.787 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:04.780+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:05.033 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:05.023+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:05.033 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:05.026+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:05.243 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:05.236+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:05.274 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:05.268+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:05.275 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:05.270+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:05.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:05.392+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:05.650 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:05.645+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:05.690 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:05.684+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:05.845 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:05.837+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:06.083 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:06.076+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:06.084 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:06.078+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:06.220 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:06.213+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:06.278 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:06.269+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:06.278 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:06.272+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:06.398 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:06.393+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:06.679 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:06.674+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:06.689 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:06.682+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:06.870 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:06.860+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:07.137 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:07.127+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:07.137 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:07.130+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:07.220 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:07.212+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:07.326 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:07.322+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:07.327 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:07.324+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:07.382 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:07.377+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:07.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:07.663+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:07.697 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:07.693+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:07.831 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:07.823+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:08.104 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:08.096+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:08.114 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:08.099+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:08.252 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:08.245+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:08.322 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:08.317+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:08.322 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:08.318+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:08.369 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:08.365+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:08.663 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:08.656+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:08.731 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:08.724+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:08.796 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:08.787+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:09.146 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:09.139+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:09.149 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:09.144+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:09.221 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:09.213+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:09.338 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:09.333+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:09.345 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:09.341+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:09.346 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:09.343+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:09.672 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:09.665+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:09.776 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:09.768+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:09.815 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:09.808+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:10.110 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:10.102+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:10.111 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:10.104+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:10.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:10.217+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:10.326 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:10.320+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:10.389 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:10.383+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:10.391 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:10.386+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:10.690 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:10.681+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:10.751 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:10.747+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:10.855 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:10.847+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:11.136 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:11.129+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:11.140 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:11.134+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:11.236 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:11.227+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:11.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:11.289+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:11.348 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:11.343+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:11.349 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:11.345+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:11.667 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:11.658+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:11.771 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:11.766+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:11.905 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:11.896+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:12.178 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:12.168+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:12.178 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:12.171+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:12.196 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:12.189+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:12.283 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:12.276+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:12.376 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:12.368+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:12.376 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:12.371+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:12.668 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:12.660+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:12.777 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:12.773+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:12.865 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:12.856+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:13.181 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:13.174+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:13.204 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:13.182+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:13.204 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:13.185+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:13.276 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:13.269+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:13.382 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:13.376+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:13.384 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:13.379+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:13.706 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:13.698+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:13.768 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:13.763+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:13.841 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:13.833+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:14.221 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:14.213+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:14.227 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:14.219+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:14.228 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:14.222+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:14.308 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:14.303+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:14.385 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:14.379+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:14.385 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:14.380+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:14.691 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:14.683+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:14.726 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:14.721+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:14.892 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:14.886+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:15.184 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:15.176+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:15.201 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:15.193+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:15.202 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:15.196+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:15.356 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:15.352+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:15.431 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:15.425+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:15.432 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:15.427+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:15.665 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:15.653+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:15.678 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:15.674+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:15.916 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:15.905+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:16.196 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:16.189+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:16.227 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:16.219+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:16.228 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:16.222+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:16.366 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:16.362+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:16.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:16.459+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:16.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:16.461+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:16.645 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:16.639+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:16.668 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:16.660+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:16.875 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:16.867+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:17.181 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:17.175+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:17.185 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:17.180+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:17.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:17.204+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:17.414 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:17.409+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:17.427 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:17.420+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:17.428 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:17.423+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:17.621 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:34:17.613+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e311 v311) 2019-12-11T22:34:17.623 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:17.618+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:17.674 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:17.659+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:17.913 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:17.906+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:18.164 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:18.158+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:18.168 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:18.162+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:18.267 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:18.258+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:18.389 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:18.385+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:18.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:18.475+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:18.483 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:18.479+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:18.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:18.623+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:18.640 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:18.636+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:18.938 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:18.928+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:19.209 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:19.201+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:19.221 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:19.204+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:19.288 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:19.281+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:19.346 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:19.341+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:19.516 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:19.509+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:19.518 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:19.512+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:19.614 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:19.608+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:19.663 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:19.655+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:19.914 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:19.907+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:20.261 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:20.253+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:20.262 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:20.256+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:20.331 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:20.324+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:20.394 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:20.388+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:20.521 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:20.515+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:20.521 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:20.517+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:20.622 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:20.616+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:20.725 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:20.700+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:20.872 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:20.864+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:21.275 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:21.266+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:21.276 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:21.269+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:21.291 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:21.283+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:21.433 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:21.428+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:21.511 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:21.503+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:21.511 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:21.506+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:21.672 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:21.667+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:21.740 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:21.733+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:21.859 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:21.849+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:22.255 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:22.248+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:22.256 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:22.250+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:22.262 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:22.256+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:22.435 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:22.430+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:22.544 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:22.537+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:22.545 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:22.540+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:22.639 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:34:22.635+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e311 v311) 2019-12-11T22:34:22.666 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:22.659+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:22.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:22.738+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:22.841 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:22.833+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:23.309 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:23.301+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:23.309 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:23.301+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:23.310 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:23.304+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:23.400 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:23.395+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:23.572 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:23.563+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:23.572 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:23.564+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:23.677 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:23.672+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:23.703 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:23.694+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:23.833 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:23.825+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:24.342 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:24.334+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:24.344 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:24.339+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:24.353 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:24.347+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:24.414 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:24.410+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:24.626 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:24.619+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:24.629 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:24.625+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:24.695 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:24.690+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:24.737 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:24.729+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:24.821 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:24.803+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:25.318 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:25.310+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:25.396 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:25.390+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:25.400 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:25.394+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:25.466 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:25.460+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:25.634 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:25.628+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:25.637 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:25.632+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:25.705 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:25.698+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:25.710 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:25.704+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:25.803 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:25.795+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:26.354 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:26.346+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:26.435 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:26.427+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:26.436 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:26.430+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:26.499 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:26.494+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:26.502 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:26.497+0000 1694f700 -1 osd.2 311 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110591 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e311) 2019-12-11T22:34:26.631 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:26.625+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:26.631 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:26.627+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:26.752 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:26.742+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:26.762 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:26.756+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:26.806 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:26.799+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:27.344 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:34:27.336+0000 1694f700 -1 osd.4 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:59.842505+0000 front 2019-12-11T22:32:59.255439+0000 (oldest deadline 2019-12-11T22:33:43.832795+0000) 2019-12-11T22:34:27.422 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:27.415+0000 1694f700 -1 osd.5 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:33:14.120504+0000 (oldest deadline 2019-12-11T22:33:54.120504+0000) 2019-12-11T22:34:27.425 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:34:27.420+0000 1694f700 -1 osd.5 311 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e309) 2019-12-11T22:34:27.527 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:27.515+0000 1694f700 -1 osd.2 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:57.132711+0000 front 2019-12-11T22:33:00.051778+0000 (oldest deadline 2019-12-11T22:33:39.978186+0000) 2019-12-11T22:34:27.527 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:27.517+0000 1694f700 -1 osd.2 311 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110591 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e311) 2019-12-11T22:34:27.645 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:27.635+0000 1694f700 -1 osd.3 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:56.851184+0000 front 2019-12-11T22:32:56.863102+0000 (oldest deadline 2019-12-11T22:33:40.213576+0000) 2019-12-11T22:34:27.645 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:27.640+0000 1694f700 -1 osd.3 311 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:27.701 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:34:27.694+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e311 v311) 2019-12-11T22:34:27.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:34:27.728+0000 1694f700 -1 osd.0 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.296992+0000 front 2019-12-11T22:32:58.312164+0000 (oldest deadline 2019-12-11T22:33:42.316040+0000) 2019-12-11T22:34:27.739 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:34:27.732+0000 1694f700 -1 osd.7 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:55.634311+0000 front 2019-12-11T22:32:55.637172+0000 (oldest deadline 2019-12-11T22:33:40.880442+0000) 2019-12-11T22:34:27.837 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:34:27.820+0000 1694f700 -1 osd.6 311 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:32:58.687205+0000 front 2019-12-11T22:32:58.683482+0000 (oldest deadline 2019-12-11T22:33:40.962193+0000) 2019-12-11T22:34:28.507 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:28.503+0000 1694f700 -1 osd.2 312 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110591 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e311) 2019-12-11T22:34:28.690 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:28.686+0000 1694f700 -1 osd.3 312 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110431 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5412555280 gen 1] snapc 0=[] ondisk+write+known_if_redirected e308) 2019-12-11T22:34:29.538 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:34:29.526+0000 1694f700 -1 osd.2 313 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110591 2.10s0 2:08a3fe50:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_a33b9fbf1f8d4caaaafa12cdceb5a30b:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e311) 2019-12-11T22:34:29.882 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:34:29.838+0000 20e25700 -1 RGWWatcher::handle_error cookie 5412555280 err (107) Transport endpoint is not connected 2019-12-11T22:34:56.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:56.472+0000 3578e700 -1 osd.3 315 build_incremental_map_msg missing incremental map 315 2019-12-11T22:34:56.503 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:56.474+0000 3578e700 -1 osd.3 315 build_incremental_map_msg missing incremental map 315 2019-12-11T22:34:56.506 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:56.475+0000 3578e700 -1 osd.3 315 build_incremental_map_msg missing incremental map 315 2019-12-11T22:34:56.867 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:56.687+0000 31f87700 -1 osd.3 315 build_incremental_map_msg missing incremental map 315 2019-12-11T22:34:57.079 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:34:56.844+0000 36f91700 -1 osd.3 315 build_incremental_map_msg missing incremental map 315 2019-12-11T22:35:34.129 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:34.117+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:35.138 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:35.131+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:36.174 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:36.168+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:37.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:37.167+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:38.218 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:38.211+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:39.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:39.205+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:40.206 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:40.200+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:41.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:41.158+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:42.145 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:42.140+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:43.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:43.133+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:44.174 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:44.150+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:45.177 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:45.171+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:46.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:46.167+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:47.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:47.170+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:48.197 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:48.193+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:49.235 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:49.230+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:50.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:50.283+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:51.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:51.297+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:52.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:52.283+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:53.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:53.266+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:54.305 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:54.300+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:55.359 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:55.354+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:56.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:56.343+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:57.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:57.388+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:57.470 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:35:57.465+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:35:58.227 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:35:58.218+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:35:58.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:58.403+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:58.461 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:35:58.456+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:35:59.104 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:35:59.099+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:35:59.232 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:35:59.216+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:35:59.373 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:35:59.368+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:35:59.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:35:59.466+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:00.093 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:00.088+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:00.187 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:00.180+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:00.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:00.331+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:00.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:00.333+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:00.515 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:00.509+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:01.103 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:01.097+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:01.131 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:01.125+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:01.220 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:01.214+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:01.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:01.322+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:01.330 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:01.324+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:01.552 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:01.546+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:01.622 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:01.616+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:01.995 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:01.987+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:02.083 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:02.078+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:02.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:02.083+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:02.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:02.205+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:02.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:02.314+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:02.322 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:02.317+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:02.510 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:02.505+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:02.660 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:02.655+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:02.956 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:02.949+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:03.044 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:03.039+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:03.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:03.094+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:03.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:03.168+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:03.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:03.309+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:03.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:03.310+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:03.483 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:03.477+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:03.677 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:03.671+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:03.921 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:03.916+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:04.053 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:04.045+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:04.147 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:04.138+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:04.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:04.167+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:04.359 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:04.352+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:04.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:04.354+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:04.437 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:04.431+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:04.635 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:04.629+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:04.935 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:04.928+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:05.069 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:05.063+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:05.117 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:05.111+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:05.218 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:05.213+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:05.404 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:05.397+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:05.404 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:05.398+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:05.446 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:05.440+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:05.677 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:05.670+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:05.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:05.930+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:06.051 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:06.039+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:06.076 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:06.070+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:06.229 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:06.219+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:06.426 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:06.420+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:06.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:06.430+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:06.437 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:06.431+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:06.718 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:06.710+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:06.893 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:06.887+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:07.032 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:07.026+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:07.067 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:07.062+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:07.265 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:07.258+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:07.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:07.396+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:07.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:07.467+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:07.474 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:07.468+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:07.692 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:07.687+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:07.925 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:07.918+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:08.005 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:08.001+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:08.060 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:08.055+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:08.298 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:08.291+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:08.368 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:08.362+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:08.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:08.468+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:08.479 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:08.470+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:08.673 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:08.667+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:08.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:08.900+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:09.036 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:09.031+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:09.093 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:09.089+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:09.282 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:09.276+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:09.341 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:09.336+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:09.505 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:09.498+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:09.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:09.500+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:09.716 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:09.712+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:09.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:09.929+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:09.999 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:09.994+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:10.086 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:10.081+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:10.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:10.291+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:10.344 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:10.338+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:10.526 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:10.519+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:10.528 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:10.523+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:10.735 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:10.731+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:10.979 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:10.971+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:11.007 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:11.002+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:11.082 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:11.076+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:11.320 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:11.316+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:11.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:11.331+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:11.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:11.565+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:11.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:11.567+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:11.776 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:11.771+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:11.943 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:11.937+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:11.986 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:11.980+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:12.057 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:12.052+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:12.298 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:12.291+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:12.323 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:12.318+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:12.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:12.554+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:12.564 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:12.558+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:12.797 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:12.791+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:12.922 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:12.915+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:13.018 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:13.013+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:13.090 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:13.086+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:13.320 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:13.314+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:13.375 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:13.363+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:13.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:13.576+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:13.583 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:13.577+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:13.840 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:13.835+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:13.973 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:13.966+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:14.057 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:14.053+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:14.065 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:14.061+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:14.308 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:14.303+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:14.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:14.397+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:14.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:14.567+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:14.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:14.570+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:14.845 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:14.840+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:15.015 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:15.008+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:15.046 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:15.038+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:15.094 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:15.081+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:15.309 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:15.302+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:15.419 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:15.414+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:15.586 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:15.580+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:15.587 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:15.581+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:15.814 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:15.805+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:16.055 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:16.050+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:16.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:16.058+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:16.068 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:16.063+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:16.283 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:16.276+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:16.405 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:16.401+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:16.640 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:16.633+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:16.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:16.634+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:16.819 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:16.814+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:17.092 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:17.086+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:17.100 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:17.095+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:17.103 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:17.097+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:17.332 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:17.326+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:17.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:17.404+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:17.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:17.596+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:17.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:17.597+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:17.832 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:17.827+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:18.095 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:18.089+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:18.130 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:18.125+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:18.139 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:18.135+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:18.332 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:18.325+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:18.432 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:18.427+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:18.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:18.624+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:18.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:18.625+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:18.785 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:18.780+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:19.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:19.119+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:19.153 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:19.147+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:19.160 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:19.156+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:19.325 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:19.319+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:19.426 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:19.423+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:19.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:19.631+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:19.640 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:19.634+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:19.746 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:19.741+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:20.084 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:20.079+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:20.173 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:20.168+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:20.190 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:20.186+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:20.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:20.321+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:20.410 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:20.405+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:20.653 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:20.642+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:20.653 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:20.643+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:20.741 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:20.735+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:21.093 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:21.086+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:21.151 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:21.146+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:21.203 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:21.197+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:21.368 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:21.362+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:21.443 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:21.439+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:21.629 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:21.623+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:21.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:21.624+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:21.792 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:21.786+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:22.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:22.081+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:22.132 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:22.126+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:22.159 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:22.152+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:22.346 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:22.340+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:22.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:22.449+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:22.681 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:22.675+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:22.682 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:22.676+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:22.834 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:22.829+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:23.112 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:23.108+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:23.129 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:23.118+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:23.131 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:23.127+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:23.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:23.329+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:23.466 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:23.458+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:23.721 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:23.714+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:23.724 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:23.718+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:23.843 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:23.839+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:24.128 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:24.123+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:24.141 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:24.135+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:24.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:24.146+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:24.357 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:24.351+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:24.434 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:24.429+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:24.688 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:24.681+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:24.689 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:24.683+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:24.848 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:24.843+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:25.098 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:25.094+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:25.140 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:25.135+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:25.184 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:25.177+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:25.360 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:25.354+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:25.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:25.434+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:25.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:25.665+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:25.674 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:25.667+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:25.882 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:25.877+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:26.053 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:26.048+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:26.101 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:26.097+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:26.226 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:26.219+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:26.327 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:26.320+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:26.430 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:26.425+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:26.658 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:26.651+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:26.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:26.653+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:26.895 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:26.890+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:27.046 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:27.042+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:27.109 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:27.104+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:27.242 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:27.235+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:27.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:27.337+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:27.418 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:27.414+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:27.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:27.610+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:27.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:27.611+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:27.869 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:27.861+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:28.043 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:28.038+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:28.095 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:28.087+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:28.209 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:28.200+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:28.370 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:28.364+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:28.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:28.395+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:28.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:28.577+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:28.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:28.577+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:28.906 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:28.901+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:29.045 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:29.040+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:29.057 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:29.052+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:29.167 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:29.161+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:29.367 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:29.361+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:29.424 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:29.420+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:29.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:29.625+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:29.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:29.627+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:29.860 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:29.855+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:30.013 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:30.008+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:30.071 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:30.067+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:30.183 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:30.176+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:30.400 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:30.393+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:30.475 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:30.470+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:30.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:30.594+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:30.602 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:30.596+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:30.824 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:30.818+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:31.041 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:31.037+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:31.088 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:31.083+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:31.188 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:31.181+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:31.406 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:31.399+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:31.518 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:31.514+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:31.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:31.592+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:31.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:31.595+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:31.785 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:31.780+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:32.064 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:32.059+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:32.096 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:32.091+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:32.196 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:32.190+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:32.436 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:32.430+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:32.481 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:32.476+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:32.561 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:32.554+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:32.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:32.555+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:32.766 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:32.761+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:33.029 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:33.024+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:33.060 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:33.055+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:33.069 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:36:33.062+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 40sec e316 v316) 2019-12-11T22:36:33.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:33.189+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:33.421 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:33.411+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:33.513 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:33.502+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:33.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:33.554+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:33.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:33.556+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:33.773 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:33.767+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:34.073 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:34.059+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:34.081 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:34.077+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:34.223 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:34.215+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:34.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:34.365+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:34.545 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:34.535+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:34.545 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:34.529+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:34.545 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:34.530+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:34.759 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:34.753+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:35.046 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:35.042+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:35.069 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:35.059+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:35.232 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:35.226+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:35.393 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:35.385+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:35.523 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:35.518+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:35.534 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:35.527+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:35.534 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:35.528+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:35.720 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:35.713+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:36.065 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:36.059+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:36.093 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:36.088+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:36.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:36.205+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:36.363 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:36.353+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:36.516 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:36.511+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:36.548 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:36.541+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:36.548 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:36.543+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:36.728 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:36.723+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:37.088 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:37.083+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:37.094 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:37.090+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:37.189 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:37.183+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:37.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:36:37.362+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.892794+0000 front 2019-12-11T22:35:18.219894+0000 (oldest deadline 2019-12-11T22:35:57.324088+0000) 2019-12-11T22:36:37.544 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:37.536+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.561277+0000 front 2019-12-11T22:35:15.461182+0000 (oldest deadline 2019-12-11T22:35:59.387078+0000) 2019-12-11T22:36:37.545 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:37.538+0000 1694f700 -1 osd.7 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:37.555 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:36:37.550+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.190867+0000 front 2019-12-11T22:35:17.021446+0000 (oldest deadline 2019-12-11T22:35:56.843414+0000) 2019-12-11T22:36:37.697 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:36:37.691+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:15.799522+0000 front 2019-12-11T22:35:17.723867+0000 (oldest deadline 2019-12-11T22:36:01.286562+0000) 2019-12-11T22:36:38.083 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:36:38.078+0000 1694f700 -1 osd.1 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:18.198421+0000 front 2019-12-11T22:35:18.197165+0000 (oldest deadline 2019-12-11T22:36:00.284651+0000) 2019-12-11T22:36:38.112 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:36:38.106+0000 1694f700 -1 osd.3 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.816434+0000 front 2019-12-11T22:35:17.861106+0000 (oldest deadline 2019-12-11T22:35:58.206474+0000) 2019-12-11T22:36:38.148 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:36:38.143+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 40sec e316 v316) 2019-12-11T22:36:38.161 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:36:38.155+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:35:17.331247+0000 front 2019-12-11T22:35:17.348161+0000 (oldest deadline 2019-12-11T22:36:01.158568+0000) 2019-12-11T22:36:38.998 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:38.863+0000 1694f700 -1 osd.7 317 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:40.056 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:36:40.010+0000 1694f700 -1 osd.7 318 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110742 2.30s0 2:0d5766c8:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.665_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e316) 2019-12-11T22:36:44.566 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:04:31.497 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.575 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:04:31.506 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.583 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:04:31.485 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.587 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:04:31.518 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.595 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:04:31.496 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.596 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:04:31.527 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.887 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:04:31.788 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.947 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:04:31.878 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:36:44.947 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:04:31.878 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:37:13.692 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:13.671+0000 3578e700 -1 osd.5 321 build_incremental_map_msg missing incremental map 321 2019-12-11T22:37:13.796 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:13.694+0000 30f85700 -1 osd.5 321 build_incremental_map_msg missing incremental map 321 2019-12-11T22:37:13.922 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:13.750+0000 3478c700 -1 osd.5 321 build_incremental_map_msg missing incremental map 321 2019-12-11T22:37:14.488 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:14.483+0000 31f87700 -1 osd.5 321 build_incremental_map_msg missing incremental map 321 2019-12-11T22:37:35.773 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:35.767+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:36.447 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:36.439+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:36.779 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:36.774+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:37.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:37.369+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:37.417 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:37.409+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:37.766 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:37.761+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:37.769 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:37.765+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:37.844 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:37.837+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:37.884 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:37.876+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:38.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:38.388+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:38.408 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:38.400+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:38.744 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:38.739+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:38.745 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:38.741+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:38.895 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:38.888+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:38.898 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:38.890+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:39.377 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:39.372+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:39.418 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:39.411+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:39.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:39.780+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:39.785 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:39.781+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:39.865 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:39.856+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:39.936 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:39.929+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:40.299 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:40.295+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:40.339 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:40.335+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:40.438 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:40.429+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:40.756 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:40.751+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:40.757 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:40.753+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:40.851 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:40.843+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:40.898 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:40.891+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:41.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:41.313+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:41.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:41.386+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:41.399 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:41.392+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:41.749 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:41.743+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:41.750 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:41.745+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:41.852 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:41.838+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:41.868 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:41.862+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:42.299 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:42.295+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:42.418 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:42.411+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:42.453 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:42.445+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:42.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:42.778+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:42.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:42.780+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:42.814 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:42.807+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:42.922 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:42.915+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:43.308 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:43.303+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:43.385 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:43.381+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:43.461 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:43.452+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:43.713 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:43.704+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:43.766 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:43.760+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:43.766 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:43.762+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:43.788 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:43.782+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:43.905 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:43.899+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:44.328 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:44.323+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:44.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:44.388+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:44.427 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:44.421+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:44.754 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:44.749+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:44.812 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:44.807+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:44.813 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:44.809+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:44.816 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:44.811+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:44.885 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:44.876+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:45.307 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:45.302+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:45.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:45.384+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:45.472 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:45.464+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:45.738 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:45.733+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:45.778 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:45.772+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:45.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:45.779+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:45.785 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:45.782+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:45.848 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:45.843+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:46.334 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:46.328+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:46.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:46.388+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:46.449 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:46.443+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:46.770 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:46.764+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:46.776 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:46.770+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:46.777 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:46.772+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:46.796 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:46.789+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:46.859 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:46.853+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:47.351 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:47.346+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:47.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:47.356+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:47.473 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:47.467+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:47.730 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:47.726+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:47.810 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:47.804+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:47.811 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:47.807+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:47.840 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:47.833+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:47.858 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:47.851+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:48.313 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:48.307+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:48.382 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:48.377+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:48.508 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:48.502+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:48.707 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:48.703+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:48.789 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:48.785+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:48.790 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:48.786+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:48.804 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:48.798+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:48.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:48.905+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:49.287 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:49.282+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:49.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:49.353+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:49.558 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:49.552+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:49.685 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:49.681+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:49.757 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:49.752+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:49.758 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:49.754+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:49.836 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:49.829+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:49.918 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:49.912+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:50.261 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:50.255+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:50.387 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:50.382+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:50.530 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:50.525+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:50.710 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:50.704+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:50.717 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:50.709+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:50.759 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:50.754+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:50.760 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:50.755+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:50.875 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:50.868+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:50.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:50.935+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:51.291 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:51.284+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:51.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:51.415+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:51.554 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:51.547+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:51.703 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:51.695+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:51.708 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:51.702+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:51.768 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:51.763+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:51.770 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:51.766+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:51.920 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:51.912+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:51.974 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:51.967+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:52.307 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:52.301+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:52.419 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:52.414+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:52.604 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:52.596+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:52.697 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:52.689+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:52.698 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:52.692+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:52.737 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:52.729+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:52.739 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:52.731+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:52.898 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:52.890+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:53.029 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:53.021+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:53.303 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:53.297+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:53.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:53.416+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:53.577 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:53.559+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:53.691 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:53.685+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:53.693 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:53.688+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:53.769 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:53.763+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:53.769 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:53.765+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:53.857 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:53.850+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:54.077 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:54.070+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:54.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:54.331+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:54.445 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:54.438+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:54.599 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:54.592+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:54.700 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:54.693+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:54.700 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:54.694+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:54.767 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:54.759+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:54.767 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:54.763+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:54.893 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:54.887+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:55.061 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:55.055+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:55.319 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:55.315+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:55.412 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:55.407+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:55.646 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:55.637+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:55.674 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:55.663+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:55.674 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:55.666+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:55.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:55.777+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:55.782 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:55.778+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:55.866 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:55.855+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:56.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:56.065+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:56.296 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:56.292+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:56.378 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:56.373+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:56.644 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:56.634+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:56.684 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:56.662+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:56.685 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:56.663+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:56.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:56.776+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:56.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:56.777+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:56.827 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:56.819+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:57.047 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:57.040+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:57.276 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:57.271+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:57.411 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:57.406+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:57.633 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:57.622+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:57.634 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:57.625+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:57.715 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:57.708+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:57.746 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:57.741+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:57.747 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:57.743+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:57.788 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:57.775+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:58.045 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:58.038+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:58.246 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:58.241+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:58.447 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:58.443+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:58.673 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:58.665+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:58.674 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:58.668+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:58.762 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:58.757+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:58.763 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:58.759+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:37:58.765 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:58.758+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:58.778 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:58.770+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:59.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:37:59.051+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:37:59.229 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:37:59.222+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:37:59.441 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:37:59.437+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:37:59.696 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:59.689+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:37:59.717 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:37:59.711+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:37:59.761 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:37:59.755+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:37:59.776 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:37:59.768+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:37:59.791 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:59.786+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:37:59.792 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:37:59.788+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:00.042 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:00.034+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:00.273 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:00.267+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:00.494 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:00.488+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:00.719 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:00.709+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:00.723 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:00.713+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:00.725 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:00.718+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:00.732 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:00.725+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:00.791 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:00.786+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:00.793 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:00.790+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:01.047 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:01.038+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:01.270 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:01.266+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:01.543 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:01.538+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:01.743 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:01.735+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:01.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:01.734+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:01.746 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:01.738+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:01.754 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:01.746+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:01.794 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:01.789+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:01.799 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:01.793+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:02.008 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:02.000+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:02.302 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:02.296+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:02.582 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:02.578+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:02.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:02.717+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:02.728 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:02.720+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:02.764 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:02.756+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:02.764 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:02.758+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:02.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:02.825+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:02.832 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:02.828+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:03.017 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:03.009+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:03.284 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:03.278+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:03.633 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:03.628+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:03.712 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:03.706+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:03.718 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:03.711+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:03.764 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:03.757+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:03.765 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:03.759+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:03.837 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:03.832+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:03.838 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:03.833+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:03.972 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:03.965+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:04.265 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:04.260+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:04.676 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:04.672+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:04.718 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:04.710+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:04.722 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:04.714+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:04.723 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:04.717+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:04.757 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:04.751+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:04.862 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:04.858+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:04.863 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:04.859+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:04.926 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:04.919+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:05.221 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:05.216+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:05.641 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:05.636+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:05.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:05.687+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:05.712 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:05.699+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:05.712 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:05.702+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:05.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:05.764+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:05.900 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:05.895+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:05.901 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:05.897+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:05.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:05.963+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:06.264 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:06.258+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:06.656 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:06.650+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:06.677 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:06.668+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:06.677 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:06.671+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:06.729 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:06.721+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:06.777 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:06.769+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:06.904 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:06.898+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:06.905 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:06.901+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:06.977 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:06.969+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:07.244 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:07.238+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:07.640 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:07.636+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:07.667 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:07.659+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:07.668 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:07.662+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:07.763 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:07.756+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:07.772 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:07.764+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:07.888 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:07.883+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:07.889 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:07.886+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:07.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:07.929+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:08.288 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:08.283+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:08.449 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:38:08.442+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e321 v321) 2019-12-11T22:38:08.596 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:08.591+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:08.662 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:08.655+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:08.664 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:08.659+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:08.782 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:08.775+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:08.792 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:08.785+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:08.896 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:08.889+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:08.940 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:08.935+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:08.940 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:08.936+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:09.316 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:09.311+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:09.630 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:09.624+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:09.689 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:09.679+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:09.689 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:09.682+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:09.788 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:09.780+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:09.823 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:09.816+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:09.869 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:09.863+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:09.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:09.926+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:09.937 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:09.931+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:10.314 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:38:10.309+0000 1694f700 -1 osd.0 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:53.536478+0000 front 2019-12-11T22:36:53.936636+0000 (oldest deadline 2019-12-11T22:37:36.837708+0000) 2019-12-11T22:38:10.618 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:10.613+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:10.730 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:10.723+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:10.732 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:10.725+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:10.795 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:10.786+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:10.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:10.849+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:10.861 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:10.854+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:10.943 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:10.937+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:10.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:10.939+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:11.582 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:11.575+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:11.754 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:11.745+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:11.754 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:11.748+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:11.816 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:11.809+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:11.831 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:11.823+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:11.836 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:11.829+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:11.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:11.895+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:11.903 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:11.898+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:12.606 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:12.601+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:12.778 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:12.769+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:12.779 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:12.772+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:12.788 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:12.781+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:12.801 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:12.794+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:12.870 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:12.864+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:12.931 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:12.924+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:12.934 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:12.929+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:13.455 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:38:13.450+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e321 v321) 2019-12-11T22:38:13.586 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:13.581+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:13.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:13.735+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:13.755 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:13.748+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:13.758 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:13.752+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:13.823 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:13.815+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:13.849 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:13.841+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:13.982 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:13.975+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:13.985 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:13.977+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:14.568 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:14.563+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:14.727 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:14.717+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:14.728 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:14.719+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:14.784 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:14.777+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:14.807 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:14.800+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:14.825 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:14.816+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:15.003 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:14.995+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:15.004 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:14.999+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:15.552 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:15.539+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:15.760 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:15.753+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:15.770 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:15.759+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:15.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:15.761+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:15.825 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:15.818+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:15.847 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:15.840+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:15.999 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:15.992+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:16.001 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:15.995+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:16.571 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:16.563+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:16.749 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:16.741+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:16.759 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:16.751+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:16.760 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:16.753+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:16.811 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:16.803+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:16.829 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:16.820+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:17.022 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:17.015+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:17.022 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:17.017+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:17.554 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:17.548+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:17.713 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:17.706+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:17.714 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:17.708+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:17.772 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:17.762+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:17.782 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:17.766+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:17.867 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:17.859+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:18.013 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:17.999+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:18.013 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:18.002+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:18.471 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:38:18.464+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e321 v321) 2019-12-11T22:38:18.544 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:18.539+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:18.696 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:18.689+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:18.699 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:18.694+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:18.809 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:18.801+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:18.813 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:18.805+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:18.888 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:18.881+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:18.975 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:18.969+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:18.977 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:18.972+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:19.596 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:19.590+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:19.749 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:19.742+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:19.749 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:19.743+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:19.782 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:19.775+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:19.857 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:19.849+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:19.895 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:19.886+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:19.995 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:19.987+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:19.996 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:19.990+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:20.580 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:20.572+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:20.779 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:20.771+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:20.783 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:20.775+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:20.784 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:20.778+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:20.827 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:38:20.819+0000 1694f700 -1 osd.4 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.498831+0000 front 2019-12-11T22:36:52.533676+0000 (oldest deadline 2019-12-11T22:37:36.112049+0000) 2019-12-11T22:38:20.878 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:20.871+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:21.038 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:21.032+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:21.039 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:21.035+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:21.541 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:21.536+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:21.794 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:21.787+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:21.810 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:21.802+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:21.810 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:21.804+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:21.877 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:21.869+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:22.037 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:22.031+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:22.038 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:22.033+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:22.580 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:38:22.574+0000 1694f700 -1 osd.2 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:54.280895+0000 front 2019-12-11T22:36:54.279379+0000 (oldest deadline 2019-12-11T22:37:40.120456+0000) 2019-12-11T22:38:22.793 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:22.785+0000 1694f700 -1 osd.5 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:37:09.542432+0000 (oldest deadline 2019-12-11T22:37:50.619929+0000) 2019-12-11T22:38:22.794 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:38:22.787+0000 1694f700 -1 osd.6 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.913729+0000 front 2019-12-11T22:36:53.353466+0000 (oldest deadline 2019-12-11T22:37:37.583504+0000) 2019-12-11T22:38:22.797 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:22.791+0000 1694f700 -1 osd.5 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:22.865 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:38:22.855+0000 1694f700 -1 osd.7 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:51.541412+0000 front 2019-12-11T22:36:52.533514+0000 (oldest deadline 2019-12-11T22:37:37.413467+0000) 2019-12-11T22:38:23.086 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:23.079+0000 1694f700 -1 osd.3 321 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:36:52.909819+0000 front 2019-12-11T22:36:52.908770+0000 (oldest deadline 2019-12-11T22:37:35.540564+0000) 2019-12-11T22:38:23.087 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:23.082+0000 1694f700 -1 osd.3 321 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:23.527 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:38:23.521+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e321 v321) 2019-12-11T22:38:23.843 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:23.837+0000 1694f700 -1 osd.5 322 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e320) 2019-12-11T22:38:24.036 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:38:24.001+0000 30f85700 -1 osd.5 322 build_incremental_map_msg missing incremental map 321 2019-12-11T22:38:24.443 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:38:24.421+0000 1694f700 -1 osd.3 322 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:110941 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5418930976 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318) 2019-12-11T22:38:26.002 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:38:25.976+0000 20e25700 -1 RGWWatcher::handle_error cookie 5418930976 err (107) Transport endpoint is not connected 2019-12-11T22:38:50.961 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:38:50.952+0000 31786700 -1 osd.1 326 build_incremental_map_msg missing incremental map 326 2019-12-11T22:38:51.406 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:38:51.391+0000 34f8d700 -1 osd.1 326 build_incremental_map_msg missing incremental map 326 2019-12-11T22:38:51.460 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:38:51.403+0000 31786700 -1 osd.1 326 build_incremental_map_msg missing incremental map 326 2019-12-11T22:38:51.985 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:38:51.976+0000 35f8f700 -1 osd.1 327 build_incremental_map_msg missing incremental map 326 2019-12-11T22:39:51.490 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:39:51.483+0000 1694f700 -1 osd.6 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:06.353228+0000 front 2019-12-11T22:39:06.357462+0000 (oldest deadline 2019-12-11T22:39:50.973784+0000) 2019-12-11T22:39:51.746 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:51.741+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:52.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:52.315+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:52.519 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:39:52.513+0000 1694f700 -1 osd.6 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:06.353228+0000 front 2019-12-11T22:39:06.357462+0000 (oldest deadline 2019-12-11T22:39:50.973784+0000) 2019-12-11T22:39:52.783 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:52.777+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:53.003 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:39:52.997+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:39:53.341 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:53.334+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:53.519 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:39:53.513+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:39:53.757 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:53.753+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:54.032 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:39:54.028+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:39:54.177 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:07:41.108 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.195 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:07:41.126 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.231 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:07:41.162 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.243 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:07:41.174 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.269 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:07:41.170 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.275 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:07:41.176 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.296 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:54.289+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:54.428 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:07:41.359 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.465 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:07:41.396 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.499 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:07:41.401 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:39:54.562 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:39:54.556+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:39:54.564 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:39:54.560+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:39:54.736 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:54.731+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:55.037 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:39:55.033+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:39:55.047 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:39:55.043+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:39:55.334 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:55.322+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:55.519 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:39:55.515+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:39:55.528 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:39:55.521+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:39:55.758 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:55.754+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:55.994 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:39:55.989+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:39:56.070 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:39:56.066+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:39:56.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:56.350+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:56.542 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:39:56.536+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:39:56.556 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:39:56.551+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:39:56.805 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:56.800+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:56.967 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:39:56.962+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:39:57.083 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:39:57.079+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:39:57.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:57.336+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:57.524 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:39:57.517+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:39:57.600 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:39:57.592+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:39:57.767 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:57.761+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:57.988 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:39:57.983+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:39:58.046 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:39:58.042+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:39:58.375 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:58.368+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:58.558 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:39:58.553+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:39:58.564 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:39:58.558+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:39:58.755 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:58.749+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:39:58.993 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:39:58.988+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:39:59.089 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:39:59.085+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:39:59.397 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:39:59.390+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:39:59.578 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:39:59.571+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:39:59.606 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:39:59.601+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:39:59.725 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:39:59.719+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:00.033 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:00.027+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:00.090 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:00.086+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:00.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:00.399+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:00.593 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:00.585+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:00.596 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:00.591+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:00.770 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:00.765+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:01.048 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:01.043+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:01.085 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:01.080+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:01.402 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:01.394+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:01.549 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:01.543+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:01.586 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:01.582+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:01.763 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:01.758+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:02.005 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:01.998+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:02.117 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:02.113+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:02.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:02.420+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:02.532 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:02.525+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:02.560 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:02.555+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:02.807 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:02.802+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:03.055 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:03.051+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:03.158 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:03.153+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:03.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:03.420+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:03.508 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:03.500+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:03.525 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:03.520+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:03.856 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:03.851+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:04.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:04.060+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:04.208 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:04.203+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:04.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:04.401+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:04.506 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:04.499+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:04.536 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:04.530+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:04.825 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:04.815+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:05.093 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:05.076+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:05.258 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:05.253+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:05.416 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:05.410+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:05.471 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:05.465+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:05.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:05.539+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:05.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:05.776+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:06.089 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:06.085+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:06.254 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:06.250+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:06.443 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:06.437+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:06.455 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:06.451+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:06.563 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:06.557+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:06.836 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:06.831+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:07.082 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:07.076+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:07.215 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:07.210+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:07.443 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:07.438+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:07.474 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:07.466+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:07.612 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:07.605+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:07.838 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:07.833+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:08.114 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:08.109+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:08.209 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:08.205+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:08.406 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:08.401+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:08.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:08.479+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:08.661 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:08.655+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:08.844 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:08.838+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:09.123 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:09.118+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:09.228 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:09.224+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:09.448 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:09.444+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:09.465 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:09.459+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:09.662 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:09.655+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:09.815 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:09.810+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:10.098 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:10.093+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:10.207 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:10.203+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:10.410 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:10.404+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:10.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:10.450+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:10.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:10.689+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:10.815 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:10.810+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:11.078 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:11.069+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:11.183 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:11.178+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:11.416 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:11.410+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:11.424 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:11.418+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:11.729 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:11.723+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:11.807 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:11.802+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:12.102 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:12.097+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:12.149 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:12.144+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:12.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:12.380+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:12.456 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:12.450+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:12.726 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:12.718+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:12.770 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:12.764+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:13.131 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:13.127+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:13.174 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:13.170+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:13.414 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:13.409+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:13.490 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:13.484+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:13.692 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:13.684+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:13.818 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:13.813+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:14.136 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:14.132+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:14.154 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:14.149+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:14.451 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:14.446+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:14.483 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:14.476+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:14.717 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:14.710+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:14.852 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:14.847+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:15.161 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:15.156+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:15.185 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:15.179+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:15.418 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:15.414+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:15.460 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:15.454+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:15.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:15.690+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:15.852 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:15.846+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:16.176 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:16.171+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:16.178 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:16.173+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:16.465 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:16.459+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:16.477 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:16.470+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:16.710 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:16.703+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:16.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:16.878+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:17.212 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:17.206+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:17.223 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:17.218+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:17.460 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:17.450+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:17.486 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:17.481+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:17.721 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:17.714+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:17.873 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:17.866+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:18.197 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:18.190+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:18.212 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:18.204+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:18.462 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:18.456+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:18.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:18.500+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:18.771 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:18.764+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:18.911 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:18.906+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:19.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:19.205+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:19.213 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:19.209+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:19.438 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:19.433+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:19.497 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:19.491+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:19.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:19.495+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:19.812 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:19.806+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:19.962 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:19.957+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:20.184 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:20.179+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:20.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:20.208+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:20.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:20.456+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:20.464 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:20.458+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:20.488 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:20.483+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:20.845 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:20.839+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:20.963 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:20.959+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:21.219 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:21.214+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:21.224 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:21.220+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:21.454 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:21.448+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:21.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:21.465+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:21.478 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:21.473+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:21.868 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:21.862+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:21.996 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:21.991+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:22.179 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:22.174+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:22.242 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:22.236+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:22.424 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:22.419+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:22.478 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:22.472+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:22.480 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:22.474+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:22.881 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:22.875+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:22.985 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:22.980+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:23.165 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:23.153+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:23.223 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:23.218+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:23.380 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:23.376+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:23.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:23.441+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:23.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:23.443+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:23.891 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:23.885+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:23.942 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:23.936+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:24.205 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:24.198+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:24.212 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:40:24.206+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e327 v327) 2019-12-11T22:40:24.224 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:24.217+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:24.353 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:24.347+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:24.480 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:24.473+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:24.481 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:24.475+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:24.865 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:24.858+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:24.910 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:24.903+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:25.207 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:25.202+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:25.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:25.226+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:25.387 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:25.381+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:25.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:25.463+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:25.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:25.470+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:25.874 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:25.867+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:25.927 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:25.921+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:26.220 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:26.214+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:26.239 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:26.233+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:26.367 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:26.358+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:26.431 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:26.424+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:26.431 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:26.425+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:26.882 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:26.876+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:26.890 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:26.885+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:27.194 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:27.188+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:27.201 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:27.196+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:27.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:27.320+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:27.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:27.457+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:27.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:27.458+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:27.839 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:27.832+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:27.926 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:27.920+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:28.160 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:28.155+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:28.199 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:28.193+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:28.311 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:28.306+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:28.434 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:28.429+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:28.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:28.429+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:28.890 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:40:28.883+0000 1694f700 -1 osd.4 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.201791+0000 front 2019-12-11T22:39:07.813655+0000 (oldest deadline 2019-12-11T22:39:52.992263+0000) 2019-12-11T22:40:28.910 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:40:28.903+0000 1694f700 -1 osd.3 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:05.517456+0000 front 2019-12-11T22:39:05.704600+0000 (oldest deadline 2019-12-11T22:39:51.378684+0000) 2019-12-11T22:40:29.186 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:40:29.180+0000 1694f700 -1 osd.0 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:08.185243+0000 front 2019-12-11T22:39:08.817940+0000 (oldest deadline 2019-12-11T22:39:52.810162+0000) 2019-12-11T22:40:29.211 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:40:29.205+0000 1694f700 -1 osd.2 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.897301+0000 front 2019-12-11T22:39:09.984131+0000 (oldest deadline 2019-12-11T22:39:54.498436+0000) 2019-12-11T22:40:29.276 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:40:29.269+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 45sec e327 v327) 2019-12-11T22:40:29.355 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:40:29.350+0000 1694f700 -1 osd.1 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.212409+0000 front 2019-12-11T22:39:08.834850+0000 (oldest deadline 2019-12-11T22:39:54.541041+0000) 2019-12-11T22:40:29.502 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:29.490+0000 1694f700 -1 osd.7 327 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:39:09.185447+0000 front 2019-12-11T22:39:09.782987+0000 (oldest deadline 2019-12-11T22:39:51.449513+0000) 2019-12-11T22:40:29.503 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:29.491+0000 1694f700 -1 osd.7 327 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:30.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:30.545+0000 1694f700 -1 osd.7 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:31.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:31.648+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:32.690 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:32.682+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:33.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:33.646+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:34.681 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:34.675+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:35.703 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:35.697+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:36.680 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:36.673+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:37.737 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:37.728+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:38.769 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:38.765+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:39.812 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:39.807+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:40.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:40.804+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:41.674 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:08:28.605 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.678 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:08:28.609 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.690 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:08:28.621 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.696 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:08:28.597 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.697 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:08:28.627 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.699 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:08:28.600 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.794 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:08:28.725 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.805 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:08:28.735 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.813 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:08:28.714 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:40:41.858 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:41.853+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:42.819 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:42.814+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:43.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:43.851+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:44.857 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:44.851+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:45.825 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:45.820+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:46.860 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:46.853+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:47.906 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:47.900+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:48.870 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:48.864+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:49.924 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:49.919+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:50.884 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:50.879+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:51.929 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:51.924+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:52.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:52.928+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:53.923 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:53.917+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:54.922 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:54.917+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:55.907 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:55.901+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:56.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:56.925+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:57.977 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:57.969+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:58.947 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:58.941+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:40:59.955 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:40:59.948+0000 1694f700 -1 osd.7 329 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:01.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:01.052+0000 1694f700 -1 osd.7 330 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:02.035 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:02.030+0000 1694f700 -1 osd.7 330 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:03.076 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:03.070+0000 1694f700 -1 osd.7 330 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:04.036 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:04.029+0000 1694f700 -1 osd.7 330 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:05.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:05.188+0000 1694f700 -1 osd.7 331 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:06.204 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:06.199+0000 1694f700 -1 osd.7 332 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:07.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:07.250+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:08.244 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:08.239+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:09.307 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:09.288+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:10.452 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:10.393+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:11.363 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:11.357+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:12.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:12.323+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:13.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:13.293+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:14.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:14.342+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:15.384 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:15.379+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:16.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:16.344+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:17.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:17.330+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:18.296 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:18.290+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:19.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:19.306+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:20.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:20.318+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:21.340 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:21.334+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:22.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:22.345+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:23.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:23.302+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:24.307 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:24.302+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:24.755 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:09:11.686 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.759 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:09:11.690 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.795 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:09:11.725 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.801 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:09:11.732 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.840 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:09:11.771 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.852 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:09:11.752 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.855 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:09:11.755 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.862 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:09:11.792 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:24.955 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:09:11.854 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:41:25.358 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:25.353+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:26.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:26.385+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:27.397 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:27.391+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:27.707 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:27.701+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:28.381 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:28.373+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:28.387 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:28.381+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:28.703 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:28.698+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:28.816 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:28.810+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:29.356 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:29.348+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:29.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:29.427+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:29.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:29.430+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:29.465 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:29.456+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:29.747 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:29.743+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:29.816 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:29.811+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:30.320 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:30.312+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:30.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:30.433+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:30.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:30.439+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:30.529 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:30.506+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:30.777 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:30.773+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:30.842 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:30.837+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:30.956 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:30.949+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:31.287 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:31.280+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:31.457 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:31.450+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:31.459 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:31.454+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:31.549 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:31.542+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:31.771 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:31.766+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:31.821 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:31.816+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:32.005 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:31.997+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:32.338 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:32.331+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:32.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:32.503+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:32.513 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:32.506+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:32.556 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:32.548+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:32.801 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:32.785+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:32.825 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:32.820+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:33.016 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:33.008+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:33.384 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:33.376+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:33.481 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:33.473+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:33.484 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:33.479+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:33.541 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:33.536+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:33.808 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:33.803+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:33.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:33.822+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:34.020 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:34.011+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:34.346 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:34.339+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:34.505 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:34.494+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:34.505 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:34.498+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:34.537 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:34.530+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:34.812 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:34.807+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:34.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:34.821+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:34.828 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:34.823+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:35.028 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:35.021+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:35.365 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:35.359+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:35.524 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:35.519+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:35.536 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:35.527+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:35.544 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:35.530+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:35.812 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:35.807+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:35.821 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:35.816+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:35.823 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:35.819+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:36.050 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:36.043+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:36.197 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:36.188+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:36.367 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:36.362+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:36.543 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:36.535+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:36.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:36.546+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:36.555 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:36.549+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:36.779 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:36.774+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:36.852 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:36.847+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:36.852 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:36.848+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:37.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:37.060+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:37.233 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:37.228+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:37.390 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:37.383+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:37.501 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:37.495+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:37.567 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:37.560+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:37.570 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:37.565+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:37.769 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:37.764+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:37.834 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:37.829+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:37.835 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:37.830+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:38.073 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:38.066+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:38.279 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:38.274+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:38.396 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:38.389+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:38.459 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:38.454+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:38.590 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:38.584+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:38.592 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:38.588+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:38.798 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:38.793+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:38.834 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:38.829+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:38.836 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:38.832+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:39.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:39.119+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:39.267 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:39.261+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:39.441 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:39.435+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:39.481 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:39.476+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:39.595 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:39.590+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:39.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:39.593+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:39.788 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:39.784+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:39.821 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:39.814+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:39.821 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:39.816+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:40.164 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:40.159+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:40.320 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:40.314+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:40.434 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:40.429+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:40.449 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:40.443+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:40.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:40.553+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:40.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:40.555+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:40.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:40.772+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:40.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:40.842+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:40.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:40.843+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:41.184 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:41.179+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:41.305 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:41.300+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:41.448 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:41.444+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:41.494 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:41.488+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:41.593 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:41.588+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:41.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:41.592+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:41.749 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:41.744+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:41.890 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:41.883+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:41.890 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:41.886+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:42.171 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:42.163+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:42.344 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:42.339+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:42.444 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:42.438+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:42.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:42.457+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:42.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:42.565+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:42.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:42.568+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:42.787 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:42.780+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:42.911 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:42.906+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:42.913 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:42.910+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:43.163 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:43.155+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:43.388 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:43.379+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:43.389 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:43.383+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:43.453 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:43.445+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:43.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:43.487+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:43.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:43.595+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:43.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:43.598+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:43.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:43.753+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:43.964 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:43.956+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:43.965 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:43.959+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:44.118 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:44.111+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:44.369 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:44.362+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:44.373 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:44.368+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:44.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:44.443+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:44.480 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:44.475+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:44.561 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:44.555+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:44.564 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:44.559+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:44.772 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:44.767+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:44.992 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:44.987+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:44.995 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:44.992+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:45.076 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:45.068+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:45.334 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:45.326+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:45.335 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:45.329+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:45.491 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:45.486+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:45.501 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:45.494+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:45.540 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:45.533+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:45.541 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:45.535+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:45.751 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:45.744+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:46.014 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:46.010+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:46.016 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:46.012+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:46.105 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:46.096+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:46.309 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:46.301+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:46.310 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:46.304+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:46.497 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:46.490+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:46.504 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:46.499+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:46.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:46.585+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:46.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:46.588+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:46.706 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:46.701+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:47.068 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:47.062+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:47.069 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:47.064+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:47.154 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:47.147+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:47.296 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:47.288+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:47.297 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:47.290+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:47.500 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:47.494+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:47.515 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:47.508+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:47.565 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:47.558+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:47.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:47.563+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:47.723 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:47.718+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:48.041 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:48.036+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:48.042 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:48.038+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:48.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:48.146+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:48.309 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:48.300+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:48.311 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:48.304+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:48.453 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:48.449+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:48.501 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:48.495+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:48.622 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:48.614+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:48.622 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:48.616+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:48.768 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:48.763+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:49.014 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:49.009+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:49.016 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:49.012+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:49.140 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:49.133+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:49.320 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:49.311+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:49.321 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:49.315+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:49.481 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:49.477+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:49.506 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:49.498+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:49.596 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:49.589+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:49.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:49.593+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:49.727 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:49.722+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:50.045 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:50.040+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:50.047 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:50.043+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:50.120 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:50.112+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:50.296 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:50.289+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:50.297 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:50.291+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:50.451 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:50.447+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:50.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:50.457+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:50.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:50.616+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:50.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:50.618+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:50.702 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:50.697+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:51.074 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:51.068+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:51.074 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:51.070+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:51.159 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:51.151+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:51.340 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:51.331+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:51.344 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:51.339+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:51.425 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:51.418+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:51.471 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:51.467+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:51.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:51.592+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:51.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:51.595+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:51.721 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:51.713+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:52.083 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:52.076+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:52.084 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:52.079+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:52.145 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:52.137+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:52.330 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:52.322+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:52.331 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:52.325+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:52.401 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:52.393+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:52.466 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:52.461+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:52.650 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:52.641+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:52.657 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:52.651+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:52.700 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:52.694+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:53.070 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:53.064+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:53.072 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:53.067+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:53.133 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:53.126+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:53.318 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:53.311+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:53.321 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:53.316+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:53.419 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:53.412+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:53.497 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:53.491+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:53.676 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:53.668+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:53.676 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:53.671+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:53.747 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:53.740+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:54.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:54.092+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:54.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:54.095+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:54.184 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:54.177+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:54.368 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:54.361+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:54.372 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:54.366+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:54.417 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:54.410+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:54.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:54.462+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:54.661 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:54.653+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:54.662 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:54.656+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:54.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:54.770+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:55.101 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:55.092+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:55.101 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:55.094+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:55.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:55.203+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:55.406 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:55.400+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:55.414 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:55.409+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:55.441 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:55.435+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:55.445 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:55.438+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:55.710 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:55.703+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:55.711 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:55.707+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:55.771 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:55.764+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:56.119 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:56.109+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:56.120 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:56.112+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:56.207 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:56.190+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:56.413 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:56.406+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:56.416 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:56.408+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:56.459 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:56.453+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:56.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:56.472+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:56.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:56.667+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:56.675 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:56.670+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:56.784 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:56.778+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:57.168 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:57.162+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:57.169 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:57.164+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:57.208 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:57.203+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:57.427 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:57.421+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:57.430 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:57.424+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:57.437 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:57.431+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:57.465 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:57.458+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:57.725 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:57.717+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:57.726 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:57.720+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:57.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:57.748+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:58.163 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:58.155+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:58.164 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:58.158+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:58.241 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:58.234+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:58.391 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:58.383+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:58.392 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:58.386+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:58.454 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:58.447+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:58.473 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:58.467+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:58.698 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:58.691+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:58.701 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:58.696+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:58.760 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:58.754+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:41:59.122 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:59.116+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:41:59.122 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:41:59.118+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:41:59.291 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:41:59.284+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:41:59.443 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:59.435+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:41:59.451 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:41:59.444+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:41:59.454 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:41:59.442+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:41:59.498 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:41:59.490+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:41:59.714 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:59.706+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:41:59.716 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:41:59.711+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:41:59.784 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:41:59.781+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 43sec e333 v333) 2019-12-11T22:41:59.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:41:59.787+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:00.152 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:00.145+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:00.153 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:00.148+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:00.340 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:00.333+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:00.478 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:00.471+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:00.481 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:00.476+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:00.497 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:00.491+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:00.504 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:00.496+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:00.755 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:00.740+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:00.755 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:00.743+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:00.759 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:00.755+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:01.152 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:01.146+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:01.153 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:01.149+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:01.343 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:01.336+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:01.439 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:01.432+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:01.442 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:01.437+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:01.471 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:01.466+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:01.490 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:01.482+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:01.761 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:01.753+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:01.761 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:01.755+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:01.807 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:01.802+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:02.188 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:02.183+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:02.188 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:02.184+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:02.375 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:02.367+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:02.466 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:02.458+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:02.466 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:02.461+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:02.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:02.473+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:02.513 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:02.505+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:02.776 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:02.769+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:02.780 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:02.774+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:02.808 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:02.802+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:03.202 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:03.198+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:03.203 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:03.199+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:03.350 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:03.342+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:03.483 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:03.478+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:03.492 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:03.484+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:03.494 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:03.487+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:03.552 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:03.544+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:03.761 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:03.753+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:03.762 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:03.756+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:03.841 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:03.834+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:04.193 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:04.186+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:04.194 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:04.189+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:04.347 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:04.340+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:04.501 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:04.495+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:04.524 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:04.517+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:04.546 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:04.539+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:04.547 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:04.541+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:04.725 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:04.718+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:04.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:04.722+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:04.802 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:42:04.798+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 43sec e333 v333) 2019-12-11T22:42:04.881 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:04.876+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:05.199 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:05.194+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:05.200 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:05.196+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:05.325 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:05.318+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:05.522 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:05.517+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:05.546 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:05.539+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:05.555 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:05.548+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:05.559 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:05.553+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:05.701 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:05.694+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:05.704 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:05.699+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:05.930 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:05.920+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:06.199 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:06.193+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:06.200 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:06.195+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:06.366 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:06.358+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:06.572 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:06.564+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:06.577 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:06.569+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:06.578 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:06.572+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:06.596 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:06.580+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:06.677 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:06.669+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:06.678 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:06.672+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:06.930 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:06.925+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:07.183 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:07.178+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:07.183 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:07.179+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:07.385 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:07.378+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:07.554 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:07.547+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:07.603 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:07.598+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:07.629 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:07.621+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:07.630 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:07.624+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:07.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:07.635+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:07.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:07.638+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:07.950 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:07.946+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:08.148 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:08.143+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:08.149 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:08.144+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:08.415 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:08.408+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:08.517 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:08.510+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:08.583 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:08.576+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:08.587 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:08.581+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:08.604 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:08.599+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:08.612 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:08.605+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:08.616 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:08.610+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:08.913 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:08.907+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:09.128 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:09.121+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:09.128 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:09.124+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:09.377 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:09.370+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:09.485 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:09.475+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:09.597 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:09.590+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:09.602 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:09.596+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:09.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:09.610+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:09.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:09.613+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:09.651 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:09.646+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:09.816 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:42:09.811+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 43sec e333 v333) 2019-12-11T22:42:09.963 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:09.955+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:10.155 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:10.148+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:10.155 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:10.151+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:10.396 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:10.388+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:10.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:10.457+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:10.613 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:10.603+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:10.615 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:10.608+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:10.618 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:10.614+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:10.636 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:10.629+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:10.638 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:10.633+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:11.009 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:11.004+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:11.187 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:11.181+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:11.188 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:11.183+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:11.433 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:11.425+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:11.455 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:11.447+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:11.601 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:11.595+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:11.607 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:11.601+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:11.612 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:11.606+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:11.682 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:11.676+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:11.683 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:11.677+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:11.970 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:11.965+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:12.218 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:12.213+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:12.219 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:12.214+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:12.401 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:12.394+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:12.460 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:12.451+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:12.557 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:12.552+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:12.619 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:12.610+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:12.619 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:12.613+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:12.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:12.636+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:12.644 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:12.638+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:13.000 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:12.995+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:13.245 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:13.240+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:13.246 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:13.241+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:13.430 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:13.423+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:13.483 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:13.475+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:13.531 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:13.526+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:13.663 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:13.652+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:13.663 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:13.653+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:13.664 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:13.655+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:13.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:13.658+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:14.049 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:42:14.044+0000 1694f700 -1 osd.0 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:45.284938+0000 front 2019-12-11T22:40:45.180816+0000 (oldest deadline 2019-12-11T22:41:28.664770+0000) 2019-12-11T22:42:14.293 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:14.288+0000 1694f700 -1 osd.3 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:44.751221+0000 front 2019-12-11T22:40:44.530318+0000 (oldest deadline 2019-12-11T22:41:27.429213+0000) 2019-12-11T22:42:14.293 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:14.289+0000 1694f700 -1 osd.3 333 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:14.444 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:42:14.436+0000 1694f700 -1 osd.4 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:46.432613+0000 front 2019-12-11T22:40:46.862398+0000 (oldest deadline 2019-12-11T22:41:30.512319+0000) 2019-12-11T22:42:14.471 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:42:14.463+0000 1694f700 -1 osd.6 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:42.760764+0000 front 2019-12-11T22:40:42.598393+0000 (oldest deadline 2019-12-11T22:41:27.854614+0000) 2019-12-11T22:42:14.513 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:42:14.508+0000 1694f700 -1 osd.2 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.480666+0000 front 2019-12-11T22:40:43.494173+0000 (oldest deadline 2019-12-11T22:41:28.743396+0000) 2019-12-11T22:42:14.635 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:14.628+0000 1694f700 -1 osd.7 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:40:43.118671+0000 front 2019-12-11T22:40:43.365627+0000 (oldest deadline 2019-12-11T22:41:28.407205+0000) 2019-12-11T22:42:14.638 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:14.633+0000 1694f700 -1 osd.7 333 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111378 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~141 in=141b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (56) in=76b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e327) 2019-12-11T22:42:14.676 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:14.669+0000 1694f700 -1 osd.5 333 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:41:02.347759+0000 (oldest deadline 2019-12-11T22:41:42.347759+0000) 2019-12-11T22:42:14.679 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:14.673+0000 1694f700 -1 osd.5 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e331) 2019-12-11T22:42:14.891 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:42:14.881+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 43sec e333 v333) 2019-12-11T22:42:15.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:15.605+0000 1694f700 -1 osd.3 334 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111497 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5415414144 gen 1] snapc 0=[] ondisk+write+known_if_redirected e330) 2019-12-11T22:42:17.076 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:42:17.061+0000 20e25700 -1 RGWWatcher::handle_error cookie 5415414144 err (107) Transport endpoint is not connected 2019-12-11T22:42:41.373 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:41.337+0000 31786700 -1 osd.3 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.378 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:41.343+0000 35f8f700 -1 osd.3 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.447 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:41.422+0000 30f85700 -1 osd.5 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.454 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:41.347+0000 31786700 -1 osd.3 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.481 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:41.364+0000 31786700 -1 osd.3 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.489 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:41.478+0000 31b86700 -1 osd.7 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.489 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:41.392+0000 36790700 -1 osd.3 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.539 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:42:41.527+0000 31f87700 -1 osd.3 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.548 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:41.520+0000 34f8d700 -1 osd.5 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.584 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:41.547+0000 34f8d700 -1 osd.5 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.724 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:41.560+0000 35b8e700 -1 osd.7 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.750 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:41.577+0000 31b86700 -1 osd.7 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.758 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:41.578+0000 31b86700 -1 osd.7 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:41.993 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:41.823+0000 3378a700 -1 osd.5 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:42.032 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:42:41.859+0000 33f8b700 -1 osd.5 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:42.156 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:42:42.128+0000 35b8e700 -1 osd.7 337 build_incremental_map_msg missing incremental map 337 2019-12-11T22:42:43.726 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:42:43.561+0000 31786700 -1 osd.1 338 build_incremental_map_msg missing incremental map 338 2019-12-11T22:42:43.736 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:42:43.602+0000 31786700 -1 osd.1 338 build_incremental_map_msg missing incremental map 338 2019-12-11T22:42:43.737 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:42:43.606+0000 31786700 -1 osd.1 338 build_incremental_map_msg missing incremental map 338 2019-12-11T22:42:44.077 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:42:44.055+0000 30f85700 -1 osd.1 338 build_incremental_map_msg missing incremental map 338 2019-12-11T22:42:44.088 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:42:44.065+0000 3378a700 -1 osd.1 338 build_incremental_map_msg missing incremental map 338 2019-12-11T22:43:41.351 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:41.346+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:41.762 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:41.755+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:42.340 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:42.336+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:42.807 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:42.801+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:43.146 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:43.139+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:43.301 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:43.297+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:43.847 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:43.841+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:44.000 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:43.994+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:44.141 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:44.134+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:44.279 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:44.274+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:44.645 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:44.638+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:44.812 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:44.806+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:44.956 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:44.949+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:44.960 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:44.956+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:45.164 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:45.157+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:45.306 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:45.299+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:45.538 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:45.534+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:45.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:45.603+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:45.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:45.848+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:45.957 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:45.951+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:45.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:45.988+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:46.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:46.192+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:46.350 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:46.345+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:46.519 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:46.515+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:46.619 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:46.615+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:46.857 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:46.850+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:46.982 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:46.977+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:46.997 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:46.990+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:47.209 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:47.203+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:47.374 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:47.370+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:47.491 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:47.486+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:47.610 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:47.605+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:47.850 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:47.843+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:47.979 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:47.960+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:48.003 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:47.997+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:48.165 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:48.158+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:48.358 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:48.354+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:48.464 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:48.459+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:48.634 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:48.629+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:48.875 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:48.869+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:48.986 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:48.983+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:49.049 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:49.042+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:49.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:49.152+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:49.370 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:49.363+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:49.485 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:49.480+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:49.600 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:49.596+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:49.893 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:49.888+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:50.005 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:50.000+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:50.040 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:50.034+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:50.161 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:50.154+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:50.341 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:50.337+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:50.526 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:50.521+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:50.569 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:50.564+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:50.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:50.938+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:50.979 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:50.974+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:51.062 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:51.056+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:51.148 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:51.142+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:51.331 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:51.326+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:51.483 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:51.479+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:51.542 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:51.537+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:51.944 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:51.939+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:51.990 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:51.983+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:52.034 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:52.027+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:52.187 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:52.180+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:52.365 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:52.361+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:52.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:52.466+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:52.511 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:52.506+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:52.985 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:52.979+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:53.014 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:53.008+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:53.022 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:53.016+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:53.143 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:53.137+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:53.412 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:53.407+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:53.464 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:53.459+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:53.467 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:53.461+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:53.939 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:53.934+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:53.987 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:53.981+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:54.045 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:54.039+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:54.135 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:54.128+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:54.405 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:54.396+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:54.429 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:54.425+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:54.497 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:54.492+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:54.984 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:54.978+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:55.024 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:55.018+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:55.040 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:55.034+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:55.133 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:55.128+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:55.369 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:55.364+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:55.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:55.416+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:55.480 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:55.475+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:55.951 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:55.946+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:55.996 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:55.989+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:56.080 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:56.073+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:56.134 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:56.128+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:56.399 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:56.394+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:56.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:56.416+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:56.518 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:56.513+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:56.998 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:56.990+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:57.010 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:57.004+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:57.041 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:57.035+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:57.124 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:57.118+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:57.405 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:57.400+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:57.454 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:57.449+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:57.532 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:57.527+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:57.963 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:57.959+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:58.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:58.012+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:58.083 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:58.076+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:58.120 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:58.113+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:58.381 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:58.376+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:58.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:58.430+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:58.527 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:58.522+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:43:58.978 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:43:58.973+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:43:59.016 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:59.010+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:43:59.128 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:43:59.122+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:43:59.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:43:59.152+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:43:59.433 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:43:59.429+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:43:59.478 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:43:59.473+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:43:59.503 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:43:59.498+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:00.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:43:59.999+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:00.026 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:00.018+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:00.141 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:00.134+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:00.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:00.176+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:00.468 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:00.463+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:00.516 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:00.512+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:00.544 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:00.540+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:00.991 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:00.987+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:01.040 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:01.032+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:01.182 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:01.174+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:01.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:01.175+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:01.463 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:01.459+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:01.473 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:01.468+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:01.502 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:01.497+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:01.967 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:01.963+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:02.080 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:02.074+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:02.217 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:02.210+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:02.231 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:02.224+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:02.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:02.460+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:02.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:02.473+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:02.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:02.502+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:02.990 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:02.985+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:03.104 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:03.098+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:03.226 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:03.219+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:03.240 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:03.233+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:03.479 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:03.475+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:03.491 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:03.485+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:03.503 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:03.499+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:03.989 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:03.984+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:04.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:04.077+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:04.194 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:04.186+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:04.286 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:04.279+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:04.454 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:04.449+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:04.492 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:04.488+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:04.530 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:04.526+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:04.982 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:04.976+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:05.096 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:05.087+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:05.165 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:05.160+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:05.263 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:05.257+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:05.445 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:05.440+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:05.489 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:05.484+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:05.532 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:05.524+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:06.006 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:06.001+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:06.121 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:06.111+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:06.148 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:06.141+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:06.229 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:06.222+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:06.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:06.452+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:06.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:06.460+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:06.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:06.575+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:07.056 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:07.049+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:07.091 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:07.084+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:07.163 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:07.156+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:07.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:07.244+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:07.427 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:07.421+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:07.478 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:07.470+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:07.573 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:07.568+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:08.082 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:08.075+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:08.112 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:08.106+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:08.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:08.161+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:08.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:08.238+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:08.420 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:08.414+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:08.452 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:08.446+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:08.597 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:08.579+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:09.051 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:09.035+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:09.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:09.121+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:09.157 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:09.150+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:09.287 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:09.281+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:09.397 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:09.390+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:09.454 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:09.449+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:09.620 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:09.615+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:10.039 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:10.032+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:10.146 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:10.128+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:10.166 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:10.160+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:10.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:10.330+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:10.391 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:10.385+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:10.413 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:10.408+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:10.664 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:10.658+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:11.077 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:11.070+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:11.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:11.110+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:11.213 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:11.207+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:11.332 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:11.325+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:11.370 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:11.364+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:11.447 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:11.443+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:11.674 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:11.670+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:12.038 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:12.034+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:12.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:12.113+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:12.215 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:12.208+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:12.361 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:12.356+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:12.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:12.355+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:12.475 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:12.468+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:12.723 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:12.718+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:13.013 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:13.009+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:13.113 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:13.106+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:13.221 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:13.214+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:13.322 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:13.315+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:13.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:13.404+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:13.488 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:13.483+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:13.685 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:13.681+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:14.053 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:14.048+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:14.069 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:14.063+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:14.188 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:14.181+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:14.296 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:14.290+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:14.402 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:14.397+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:14.450 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:14.446+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:14.665 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:14.660+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:15.035 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:15.025+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:15.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:15.045+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:15.240 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:15.233+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:15.297 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:44:15.288+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 42sec e338 v338) 2019-12-11T22:44:15.345 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:15.338+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:15.392 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:15.385+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:15.446 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:15.441+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:15.633 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:15.629+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:16.017 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:16.013+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:16.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:16.013+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:16.250 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:16.243+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:16.310 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:16.303+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:16.440 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:16.436+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:16.482 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:16.472+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:16.644 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:16.640+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:16.983 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:16.974+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:17.061 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:17.055+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:17.277 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:17.271+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:17.295 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:17.288+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:17.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:17.434+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:17.486 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:17.481+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:17.634 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:17.629+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:18.014 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:18.008+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:18.081 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:18.076+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:18.308 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:18.302+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:18.321 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:18.314+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:18.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:18.401+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:18.463 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:18.459+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:18.648 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:18.643+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:18.987 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:18.980+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:19.067 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:19.063+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:19.312 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:19.307+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:19.369 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:19.364+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:19.421 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:19.416+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:19.448 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:44:19.444+0000 1694f700 -1 osd.0 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.830385+0000 front 2019-12-11T22:43:02.826868+0000 (oldest deadline 2019-12-11T22:43:44.949971+0000) 2019-12-11T22:44:19.635 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:44:19.631+0000 1694f700 -1 osd.2 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.555815+0000 front 2019-12-11T22:43:02.561505+0000 (oldest deadline 2019-12-11T22:43:43.618141+0000) 2019-12-11T22:44:19.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:19.964+0000 1694f700 -1 osd.7 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.152584+0000 front 2019-12-11T22:43:01.258395+0000 (oldest deadline 2019-12-11T22:43:41.202544+0000) 2019-12-11T22:44:20.094 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:44:20.089+0000 1694f700 -1 osd.1 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:03.261237+0000 front 2019-12-11T22:43:01.647357+0000 (oldest deadline 2019-12-11T22:43:44.516628+0000) 2019-12-11T22:44:20.311 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:44:20.304+0000 1694f700 -1 osd.4 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:00.120187+0000 front 2019-12-11T22:43:02.527127+0000 (oldest deadline 2019-12-11T22:43:42.322745+0000) 2019-12-11T22:44:20.360 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:44:20.348+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 42sec e338 v338) 2019-12-11T22:44:20.440 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:44:20.435+0000 1694f700 -1 osd.6 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:02.089980+0000 front 2019-12-11T22:43:02.174196+0000 (oldest deadline 2019-12-11T22:43:43.769609+0000) 2019-12-11T22:44:20.444 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:44:20.435+0000 1694f700 -1 osd.3 338 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:43:01.262958+0000 front 2019-12-11T22:42:57.495013+0000 (oldest deadline 2019-12-11T22:43:41.231574+0000) 2019-12-11T22:44:37.681 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:12:24.612 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.685 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:12:24.617 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.696 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:12:24.627 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.699 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:12:24.601 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.702 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:12:24.633 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.708 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:12:24.609 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.894 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:12:24.826 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.922 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:12:24.823 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:37.941 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:12:24.871 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:44:50.611 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:50.602+0000 1694f700 -1 osd.7 340 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:51.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:51.619+0000 1694f700 -1 osd.7 340 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:52.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:52.576+0000 1694f700 -1 osd.7 340 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:53.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:53.598+0000 1694f700 -1 osd.7 341 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:54.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:54.593+0000 1694f700 -1 osd.7 341 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:55.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:55.586+0000 1694f700 -1 osd.7 341 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:56.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:56.575+0000 1694f700 -1 osd.7 341 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:57.551 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:57.546+0000 1694f700 -1 osd.7 341 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:44:58.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:44:58.596+0000 1694f700 -1 osd.7 341 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:00.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:00.130+0000 1694f700 -1 osd.7 342 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:01.103 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:01.085+0000 1694f700 -1 osd.7 343 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:02.137 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:02.132+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:03.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:03.176+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:04.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:04.182+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:05.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:05.221+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:06.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:06.222+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:07.229 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:07.223+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:08.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:08.273+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:09.240 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:09.234+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:10.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:10.289+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:11.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:11.246+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:12.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:12.289+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:13.304 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:13.298+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:14.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:14.267+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:15.293 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:15.288+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:16.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:16.296+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:17.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:17.344+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:17.693 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:17.688+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:18.247 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:18.240+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:18.377 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:18.369+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:18.378 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:18.371+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:18.677 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:18.671+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:19.274 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:19.269+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:19.341 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:19.334+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:19.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:19.338+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:19.418 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:19.413+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:19.688 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:19.683+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:19.895 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:19.888+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:20.301 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:20.296+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:20.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:20.382+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:20.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:20.384+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:20.442 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:20.436+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:20.703 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:20.697+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:20.881 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:20.874+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:21.287 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:21.280+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:21.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:21.348+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:21.358 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:21.352+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:21.458 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:21.453+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:21.707 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:21.701+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:21.860 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:21.852+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:22.309 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:22.304+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:22.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:22.316+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:22.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:22.321+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:22.486 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:22.481+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:22.678 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:22.668+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:22.911 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:22.902+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:22.938 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:22.929+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:23.312 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:23.305+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:23.312 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:23.308+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:23.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:23.329+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:23.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:23.330+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:23.511 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:23.506+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:23.666 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:23.661+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:23.922 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:23.913+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:23.967 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:23.955+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:24.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:24.332+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:24.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:24.338+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:24.355 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:24.350+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:24.355 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:24.351+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:24.539 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:24.534+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:24.672 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:24.667+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:24.944 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:24.931+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:24.969 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:24.960+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:25.337 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:25.332+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:25.337 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:25.333+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:25.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:25.341+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:25.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:25.346+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:25.583 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:25.578+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:25.692 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:25.688+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:25.950 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:25.943+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:25.956 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:25.948+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:26.297 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:26.293+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:26.300 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:26.297+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:26.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:26.387+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:26.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:26.390+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:26.618 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:26.604+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:26.667 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:26.652+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:26.945 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:26.937+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:27.006 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:26.999+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:27.335 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:27.331+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:27.338 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:27.334+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:27.382 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:27.375+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:27.383 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:27.378+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:27.633 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:27.628+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:27.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:27.703+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:27.951 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:27.944+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:27.982 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:27.976+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:28.377 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:28.371+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:28.377 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:28.373+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:28.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:28.382+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:28.396 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:28.391+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:28.623 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:28.617+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:28.670 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:28.665+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:28.992 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:28.986+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:28.995 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:28.989+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:29.364 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:29.358+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:29.364 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:29.360+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:29.366 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:29.360+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:29.370 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:29.365+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:29.629 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:29.625+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:29.659 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:29.655+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:29.973 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:29.967+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:29.987 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:29.981+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:30.368 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:30.362+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:30.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:30.367+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:30.387 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:30.380+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:30.387 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:30.383+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:30.509 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:30.504+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:30.615 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:30.610+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:30.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:30.692+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:30.950 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:30.943+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:31.017 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:31.011+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:31.377 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:31.370+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:31.378 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:31.373+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:31.400 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:31.394+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:31.401 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:31.396+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:31.565 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:31.554+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:31.657 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:31.651+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:31.670 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:31.663+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:31.920 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:31.913+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:32.024 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:32.015+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:32.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:32.325+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:32.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:32.328+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:32.439 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:32.434+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:32.439 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:32.436+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:32.539 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:32.534+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:32.619 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:32.613+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:32.682 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:32.678+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:32.947 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:32.942+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:33.046 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:33.040+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:33.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:33.353+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:33.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:33.355+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:33.454 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:33.447+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:33.455 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:33.450+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:33.578 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:33.573+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:33.619 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:33.613+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:33.702 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:33.697+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:33.971 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:33.962+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:34.010 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:34.002+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:34.433 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:34.405+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:34.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:34.408+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:34.490 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:34.482+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:34.491 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:34.485+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:34.566 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:34.560+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:34.668 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:34.661+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:34.684 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:34.678+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:34.960 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:34.953+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:35.051 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:35.043+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:35.452 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:35.444+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:35.453 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:35.447+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:35.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:35.463+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:35.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:35.465+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:35.557 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:35.551+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:35.645 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:35.639+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:35.676 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:35.670+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:36.010 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:36.003+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:36.031 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:36.024+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:36.454 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:36.446+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:36.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:36.448+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:36.505 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:36.497+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:36.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:36.499+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:36.581 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:36.574+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:36.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:36.635+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:36.686 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:36.680+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:37.006 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:36.999+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:37.066 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:37.058+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:37.468 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:37.463+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:37.471 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:37.466+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:37.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:37.464+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:37.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:37.467+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:37.621 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:37.613+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:37.622 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:37.616+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:37.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:37.635+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:37.688 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:37.683+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:37.991 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:37.983+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:38.027 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:38.020+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:38.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:38.435+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:38.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:38.439+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:38.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:38.459+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:38.466 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:38.461+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:38.624 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:38.618+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:38.630 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:38.622+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:38.631 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:38.624+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:38.693 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:38.687+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:38.967 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:38.960+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:39.045 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:39.038+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:39.416 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:39.408+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:39.417 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:39.409+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:39.472 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:39.467+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:39.475 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:39.470+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:39.576 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:39.572+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:39.611 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:39.601+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:39.612 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:39.606+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:39.692 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:39.687+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:39.934 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:39.927+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:40.028 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:40.020+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:40.421 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:40.413+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:40.422 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:40.415+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:40.503 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:40.497+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:40.504 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:40.499+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:40.605 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:40.599+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:40.629 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:40.622+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:40.636 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:40.628+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:40.718 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:40.714+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:40.969 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:40.957+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:41.045 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:41.039+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:41.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:41.442+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:41.451 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:41.446+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:41.510 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:41.505+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:41.510 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:41.506+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:41.591 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:41.586+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:41.677 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:41.669+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:41.678 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:41.672+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:41.763 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:41.759+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:41.938 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:41.930+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:42.080 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:42.073+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:42.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:42.401+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:42.412 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:42.406+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:42.463 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:42.459+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:42.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:42.461+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:42.615 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:42.610+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:42.730 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:42.723+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:42.734 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:42.728+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:42.810 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:42.805+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:42.960 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:42.952+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:43.056 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:43.048+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:43.402 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:43.394+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:43.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:43.397+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:43.470 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:43.464+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:43.471 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:43.466+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:43.604 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:43.599+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:43.784 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:43.777+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:43.787 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:43.782+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:43.831 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:43.826+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:43.957 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:43.949+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:44.050 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:44.043+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:44.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:44.408+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:44.422 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:44.417+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:44.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:44.464+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:44.472 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:44.468+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:44.631 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:44.626+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:44.795 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:44.788+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:44.796 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:44.790+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:44.879 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:44.875+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:45.003 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:44.996+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:45.013 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:45.005+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:45.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:45.402+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:45.410 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:45.404+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:45.477 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:45.472+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:45.478 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:45.473+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:45.585 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:45.580+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:45.839 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:45.826+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:45.839 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:45.831+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:45.842 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:45.837+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:45.984 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:45.978+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:46.012 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:46.005+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:46.420 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:46.413+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:46.421 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:46.415+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:46.445 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:46.440+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:46.446 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:46.442+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:46.562 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:46.558+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:46.816 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:46.808+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:46.817 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:46.811+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:46.837 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:46.833+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:46.985 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:46.974+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:46.994 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:46.986+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:47.406 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:47.398+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:47.407 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:47.401+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:47.495 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:47.490+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:47.496 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:47.491+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:47.556 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:47.547+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:47.779 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:47.771+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:47.780 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:47.774+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:47.813 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:47.808+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:48.004 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:47.997+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:48.011 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:48.004+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:48.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:48.439+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:48.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:48.442+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:48.540 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:48.535+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:48.543 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:48.539+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:48.556 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:48.547+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:48.775 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:48.768+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:48.775 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:48.769+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:48.784 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:48.779+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:49.040 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:49.033+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:49.057 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:49.051+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:49.416 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:49.409+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:49.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:49.414+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:49.584 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:49.579+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:49.585 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:49.580+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:49.597 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:49.593+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:49.735 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:49.727+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:49.735 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:49.730+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:49.748 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:49.741+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:50.073 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:50.065+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:50.085 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:50.077+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:50.397 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:50.389+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:50.398 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:50.392+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:50.611 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:50.606+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:50.630 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:50.625+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:50.632 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:50.628+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:50.713 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:50.707+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:50.756 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:50.749+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:50.757 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:50.753+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:51.048 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:51.041+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:51.053 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:51.047+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:51.433 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:51.425+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:51.433 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:51.428+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:51.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:51.569+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:51.641 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:51.634+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:51.641 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:51.637+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:51.750 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:51.745+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:51.790 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:51.783+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:51.794 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:51.788+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:52.013 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:52.004+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:52.021 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:52.014+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:52.423 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:52.415+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:52.424 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:52.417+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:52.535 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:52.530+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:52.602 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:52.597+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:52.603 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:52.598+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:52.776 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:52.770+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:52.789 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:52.781+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:52.790 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:52.784+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:53.026 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:53.018+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:53.045 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:53.038+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:53.442 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:53.435+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:53.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:53.440+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:53.565 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:53.560+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:53.566 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:53.562+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:53.572 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:53.567+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:53.768 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:53.760+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:53.772 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:53.766+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:53.821 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:53.816+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:54.032 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:54.023+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:54.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:54.030+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:54.460 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:54.453+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:54.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:54.458+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:54.536 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:54.530+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:54.560 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:54.554+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:54.560 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:54.556+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:54.778 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:54.770+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:54.779 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:54.773+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:54.823 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:54.818+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:55.049 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:55.042+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:55.061 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:55.053+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:55.517 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:55.506+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:55.520 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:55.512+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:55.549 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:55.544+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:55.550 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:55.546+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:55.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:55.568+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:55.666 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:45:55.660+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e344 v344) 2019-12-11T22:45:55.748 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:55.740+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:55.748 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:55.742+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:55.778 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:55.773+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:56.057 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:56.049+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:56.070 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:56.063+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:56.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:56.509+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:56.516 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:56.511+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:56.579 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:56.559+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:56.582 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:56.577+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:56.586 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:56.580+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:56.786 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:56.778+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:56.787 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:56.781+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:56.828 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:56.824+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:57.108 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:57.101+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:57.111 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:57.103+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:57.516 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:57.512+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:57.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:57.513+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:57.551 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:57.541+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:57.555 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:57.543+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:57.576 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:57.572+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:57.788 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:57.780+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:57.790 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:57.782+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:57.830 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:57.824+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:58.144 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:58.138+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:58.162 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:58.156+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:58.528 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:58.524+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:58.529 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:58.525+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:58.541 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:58.534+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:58.544 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:58.538+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:58.626 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:58.620+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:58.761 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:58.753+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:58.764 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:58.759+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:58.873 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:58.868+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:45:59.111 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:45:59.104+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:45:59.139 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:45:59.132+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:45:59.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:59.500+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:45:59.510 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:45:59.504+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:45:59.525 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:59.520+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:45:59.528 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:45:59.524+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:45:59.633 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:45:59.628+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:45:59.819 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:59.810+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:45:59.819 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:45:59.812+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:45:59.890 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:45:59.884+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:00.166 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:00.159+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:00.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:00.166+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:00.498 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:00.490+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:00.499 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:00.493+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:00.508 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:00.504+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:00.510 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:00.505+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:00.668 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:00.664+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:00.681 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:46:00.676+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e344 v344) 2019-12-11T22:46:00.861 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:00.856+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:00.863 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:00.857+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:00.866 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:00.860+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:01.156 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:01.148+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:01.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:01.151+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:01.531 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:01.521+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:01.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:01.525+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:01.541 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:01.537+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:01.542 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:01.538+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:01.639 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:01.633+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:01.826 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:01.822+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:01.827 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:01.819+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:01.828 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:01.822+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:02.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:02.166+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:02.193 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:02.187+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:02.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:02.498+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:02.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:02.500+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:02.546 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:02.538+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:02.547 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:02.541+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:02.602 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:02.597+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:02.790 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:02.786+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:02.859 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:02.852+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:02.863 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:02.857+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:03.198 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:03.190+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:03.214 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:03.207+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:03.478 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:03.470+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:03.478 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:03.472+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:03.586 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:03.581+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:03.587 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:03.583+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:03.614 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:03.609+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:03.821 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:03.816+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:03.902 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:03.894+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:03.903 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:03.897+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:04.179 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:04.172+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:04.191 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:04.185+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:04.484 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:04.477+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:04.485 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:04.479+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:04.626 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:04.621+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:04.627 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:04.623+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:04.638 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:04.634+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:04.868 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:04.863+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:04.918 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:04.912+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:04.918 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:04.913+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:05.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:05.146+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:05.227 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:05.219+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:05.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:05.440+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:05.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:05.442+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:05.616 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:05.610+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:05.617 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:05.612+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:05.658 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:05.654+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:05.698 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:46:05.692+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e344 v344) 2019-12-11T22:46:05.913 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:05.909+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:05.919 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:05.912+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:05.931 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:05.918+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:06.116 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:06.109+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:06.193 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:06.186+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:06.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:06.420+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:06.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:06.423+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:06.571 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:06.564+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:06.574 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:06.567+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:06.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:06.635+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:06.875 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:06.871+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:06.931 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:06.923+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:06.931 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:06.926+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:07.170 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:07.159+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:07.177 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:07.167+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:07.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:07.405+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:07.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:07.408+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:07.593 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:07.589+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:07.606 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:07.601+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:07.607 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:07.603+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:07.855 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:07.850+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:07.967 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:07.960+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:07.971 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:07.965+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:08.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:08.203+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:08.219 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:08.211+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:08.387 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:08.379+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:08.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:08.382+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:08.624 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:08.619+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:08.661 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:08.650+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:08.661 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:08.652+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:08.827 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:08.822+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:09.014 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:09.006+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:09.015 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:09.009+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:09.176 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:09.169+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:09.233 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:09.226+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:09.387 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:09.380+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:09.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:09.382+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:09.600 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:09.594+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:09.669 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:09.663+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:09.669 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:09.665+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:09.880 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:46:09.874+0000 1694f700 -1 osd.2 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:32.794439+0000 front 2019-12-11T22:44:32.722801+0000 (oldest deadline 2019-12-11T22:45:18.610550+0000) 2019-12-11T22:46:10.013 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:10.005+0000 1694f700 -1 osd.5 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:44:57.387928+0000 (oldest deadline 2019-12-11T22:45:37.387928+0000) 2019-12-11T22:46:10.015 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:10.008+0000 1694f700 -1 osd.5 344 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:10.216 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:10.208+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:10.240 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:46:10.230+0000 1694f700 -1 osd.6 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.137861+0000 front 2019-12-11T22:44:35.145128+0000 (oldest deadline 2019-12-11T22:45:19.804529+0000) 2019-12-11T22:46:10.396 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:10.389+0000 1694f700 -1 osd.7 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:35.756331+0000 front 2019-12-11T22:44:35.454922+0000 (oldest deadline 2019-12-11T22:45:17.735677+0000) 2019-12-11T22:46:10.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:46:10.393+0000 1694f700 -1 osd.7 344 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:111838 2.8s0 2:114f52a2:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_88c16cb1ca844823b34b994ff2e3bbe2:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e338) 2019-12-11T22:46:10.586 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:46:10.581+0000 1694f700 -1 osd.0 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:31.879557+0000 front 2019-12-11T22:44:32.154521+0000 (oldest deadline 2019-12-11T22:45:17.677410+0000) 2019-12-11T22:46:10.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:10.714+0000 1694f700 -1 osd.3 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:37.127365+0000 front 2019-12-11T22:44:36.979010+0000 (oldest deadline 2019-12-11T22:45:17.356528+0000) 2019-12-11T22:46:10.723 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:10.716+0000 1694f700 -1 osd.3 344 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:10.756 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:46:10.749+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e344 v344) 2019-12-11T22:46:11.173 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:46:11.123+0000 1694f700 -1 osd.5 345 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e342) 2019-12-11T22:46:11.233 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:46:11.227+0000 1694f700 -1 osd.4 344 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:44:36.653979+0000 front 2019-12-11T22:44:36.725670+0000 (oldest deadline 2019-12-11T22:45:22.534911+0000) 2019-12-11T22:46:11.770 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:46:11.764+0000 1694f700 -1 osd.3 345 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:111895 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 900729776 gen 1] snapc 0=[] ondisk+write+known_if_redirected e340) 2019-12-11T22:46:13.151 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:46:13.114+0000 20e25700 -1 RGWWatcher::handle_error cookie 900729776 err (107) Transport endpoint is not connected 2019-12-11T22:46:31.161 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:46:31.153+0000 124b0700 -1 osd.1 349 build_incremental_map_msg missing incremental map 349 2019-12-11T22:46:31.280 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:46:31.233+0000 34f8d700 -1 osd.1 349 build_incremental_map_msg missing incremental map 349 2019-12-11T22:46:31.414 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:46:31.334+0000 32788700 -1 osd.1 349 build_incremental_map_msg missing incremental map 349 2019-12-11T22:47:30.882 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:30.875+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:30.884 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:30.879+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:31.242 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:31.238+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:31.409 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:31.401+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:31.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:31.792+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:31.920 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:31.914+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:31.928 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:31.924+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:32.247 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:32.241+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:32.340 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:32.335+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:32.432 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:32.425+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:32.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:32.795+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:32.885 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:32.878+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:32.969 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:32.963+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:33.280 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:33.275+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:33.299 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:33.295+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:33.393 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:33.387+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:33.770 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:33.761+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:33.860 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:33.854+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:34.004 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:33.997+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:34.302 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:34.297+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:34.347 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:34.343+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:34.348 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:34.344+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:34.366 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:34.360+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:34.740 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:34.734+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:34.869 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:34.857+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:34.984 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:34.979+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:35.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:35.325+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:35.357 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:35.351+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:35.367 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:35.359+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:35.383 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:35.371+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:35.701 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:35.694+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:35.826 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:35.819+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:36.033 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:36.027+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:36.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:36.328+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:36.354 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:36.346+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:36.375 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:36.369+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:36.395 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:36.390+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:36.703 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:36.697+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:36.861 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:36.855+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:37.020 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:37.016+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:37.385 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:37.381+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:37.394 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:37.390+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:37.402 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:37.398+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:37.431 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:37.425+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:37.702 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:37.695+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:37.897 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:37.884+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:37.984 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:37.979+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:38.345 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:38.339+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:38.349 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:38.344+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:38.427 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:38.421+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:38.451 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:38.444+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:38.700 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:38.694+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:38.886 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:38.873+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:38.953 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:38.949+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:39.336 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:39.326+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:39.375 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:39.370+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:39.477 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:39.471+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:39.479 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:39.475+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:39.665 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:39.657+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:39.907 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:39.900+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:40.000 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:39.995+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:40.361 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:40.356+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:40.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:40.385+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:40.480 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:40.473+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:40.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:40.513+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:40.692 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:40.687+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:40.938 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:40.933+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:41.029 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:41.019+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:41.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:41.320+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:41.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:41.401+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:41.479 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:41.473+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:41.539 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:41.531+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:41.746 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:41.739+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:41.910 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:41.902+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:42.060 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:42.056+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:42.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:42.320+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:42.452 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:42.448+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:42.484 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:42.479+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:42.563 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:42.558+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:42.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:42.721+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:42.880 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:42.874+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:43.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:43.103+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:43.338 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:43.334+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:43.477 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:43.473+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:43.490 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:43.483+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:43.577 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:43.572+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:43.693 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:43.687+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:43.932 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:43.923+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:44.107 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:44.103+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:44.298 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:44.293+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:44.497 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:44.492+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:44.528 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:44.522+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:44.607 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:44.601+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:44.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:44.653+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:44.898 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:44.892+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:45.068 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:45.063+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:45.286 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:45.280+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:45.484 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:45.478+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:45.566 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:45.560+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:45.595 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:45.590+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:45.675 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:45.669+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:45.941 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:45.934+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:46.029 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:46.024+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:46.328 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:46.322+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:46.461 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:46.454+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:46.565 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:46.560+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:46.623 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:46.616+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:46.716 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:46.706+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:46.939 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:46.933+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:47.026 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:47.019+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:47.292 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:47.287+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:47.419 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:47.412+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:47.561 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:47.552+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:47.616 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:47.611+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:47.750 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:47.744+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:47.968 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:47.960+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:48.033 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:48.027+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:48.282 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:48.275+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:48.405 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:48.393+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:48.568 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:48.562+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:48.639 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:48.633+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:48.738 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:48.731+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:48.940 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:48.934+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:49.063 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:49.056+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:49.281 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:49.275+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:49.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:49.402+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:49.526 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:49.520+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:49.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:49.648+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:49.759 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:49.735+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:49.921 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:49.914+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:50.042 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:50.037+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:50.264 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:50.259+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:50.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:50.384+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:50.517 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:50.508+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:50.644 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:50.638+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:50.796 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:50.788+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:50.797 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:50.790+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:50.969 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:50.959+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:51.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:51.087+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:51.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:51.304+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:51.366 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:51.361+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:51.510 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:51.503+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:51.661 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:51.656+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:51.756 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:51.744+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:51.756 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:51.745+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:51.993 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:51.986+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:52.137 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:52.131+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:52.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:52.304+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:52.387 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:52.382+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:52.518 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:52.510+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:52.651 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:52.646+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:52.707 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:52.702+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:52.713 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:52.704+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:52.987 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:52.980+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:53.147 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:53.142+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:53.334 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:53.327+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:53.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:53.410+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:53.476 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:53.469+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:53.666 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:53.662+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:53.715 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:53.709+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:53.718 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:53.712+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:53.974 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:53.968+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:54.186 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:54.181+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:54.393 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:54.380+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:54.410 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:54.406+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:54.442 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:54.435+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:54.705 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:54.700+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:54.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:54.742+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:54.750 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:54.744+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:54.947 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:54.940+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:55.209 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:55.204+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:55.369 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:55.363+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:55.425 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:55.420+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:55.460 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:55.452+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:55.750 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:55.744+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:55.779 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:55.771+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:55.786 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:55.772+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:55.923 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:55.917+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:56.225 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:56.220+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:56.337 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:56.332+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:56.449 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:56.443+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:56.476 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:56.471+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:56.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:56.779+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:56.816 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:56.811+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:56.817 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:56.812+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:56.965 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:56.958+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:57.182 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:57.177+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:57.361 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:57.355+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:57.484 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:57.473+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:57.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:57.499+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:57.791 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:57.785+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:57.792 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:57.787+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:57.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:57.789+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:58.007 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:58.000+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:58.238 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:58.233+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:58.348 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:58.342+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:58.453 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:58.440+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:58.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:58.499+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:58.807 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:58.799+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:58.807 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:58.801+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:58.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:58.837+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:58.973 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:58.966+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:47:59.200 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:47:59.195+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:47:59.389 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:47:59.385+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:47:59.433 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:47:59.426+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:47:59.502 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:47:59.491+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:47:59.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:59.800+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:47:59.810 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:47:59.802+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:47:59.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:47:59.822+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:47:59.967 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:47:59.960+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:00.247 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:00.242+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:00.362 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:00.357+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:00.433 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:00.419+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:00.521 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:00.516+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:00.786 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:00.779+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:00.786 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:00.780+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:00.823 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:00.818+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:00.986 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:00.979+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:01.260 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:01.256+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:01.393 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:01.388+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:01.402 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:01.396+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:01.474 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:01.470+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:01.818 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:01.812+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:01.819 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:01.813+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:01.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:01.870+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:01.997 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:01.990+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:02.219 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:02.215+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:02.364 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:02.359+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:02.430 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:02.423+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:02.491 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:02.485+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:02.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:02.796+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:02.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:02.799+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:02.879 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:02.872+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:02.972 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:02.965+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:03.185 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:03.168+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:03.327 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:03.315+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:03.453 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:03.446+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:03.526 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:03.520+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:03.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:03.827+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:03.845 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:03.838+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:03.845 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:03.840+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:03.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:03.930+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:04.193 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:04.188+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:04.351 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:04.337+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:04.411 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:04.404+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:04.556 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:04.550+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:04.811 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:04.805+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:04.862 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:04.855+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:04.862 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:04.856+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:04.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:04.900+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:05.218 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:05.213+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:05.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:05.320+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:05.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:05.395+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:05.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:05.569+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:05.798 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:05.793+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:05.862 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:05.854+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:05.905 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:05.898+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:05.905 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:05.899+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:06.061 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:48:06.048+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 43sec e349 v349) 2019-12-11T22:48:06.255 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:06.251+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:06.358 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:06.354+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:06.437 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:06.431+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:06.538 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:06.533+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:06.756 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:06.752+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:06.845 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:06.838+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:06.932 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:06.925+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:06.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:06.932+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:07.269 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:07.261+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:07.364 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:07.360+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:07.432 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:07.408+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:07.551 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:07.545+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:07.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:07.778+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:07.810 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:07.804+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:07.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:07.904+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:07.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:07.905+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:08.273 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:08.268+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:08.368 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:08.355+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:08.462 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:08.454+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:08.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:08.506+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:08.763 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:08.758+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:08.787 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:08.781+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:08.874 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:08.866+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:08.878 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:08.867+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:09.317 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:09.312+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:09.380 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:09.375+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:09.481 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:09.473+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:09.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:09.544+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:09.720 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:09.715+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:09.789 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:09.782+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:09.871 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:09.865+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:09.872 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:09.867+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:10.285 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:48:10.278+0000 1694f700 -1 osd.2 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.791121+0000 front 2019-12-11T22:46:48.808470+0000 (oldest deadline 2019-12-11T22:47:30.617725+0000) 2019-12-11T22:48:10.354 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:48:10.347+0000 1694f700 -1 osd.1 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.809080+0000 front 2019-12-11T22:46:49.596986+0000 (oldest deadline 2019-12-11T22:47:33.997021+0000) 2019-12-11T22:48:10.466 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:48:10.460+0000 1694f700 -1 osd.6 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:48.033861+0000 front 2019-12-11T22:46:48.007183+0000 (oldest deadline 2019-12-11T22:47:30.659205+0000) 2019-12-11T22:48:10.586 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:48:10.580+0000 1694f700 -1 osd.0 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.131983+0000 front 2019-12-11T22:46:47.606747+0000 (oldest deadline 2019-12-11T22:47:30.950762+0000) 2019-12-11T22:48:10.754 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:48:10.750+0000 1694f700 -1 osd.3 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:49.595447+0000 front 2019-12-11T22:46:50.644168+0000 (oldest deadline 2019-12-11T22:47:32.054001+0000) 2019-12-11T22:48:10.786 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:48:10.774+0000 1694f700 -1 osd.4 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:47.190376+0000 front 2019-12-11T22:46:47.174059+0000 (oldest deadline 2019-12-11T22:47:30.361295+0000) 2019-12-11T22:48:10.829 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:10.822+0000 1694f700 -1 osd.7 349 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:46:45.542240+0000 front 2019-12-11T22:46:45.570840+0000 (oldest deadline 2019-12-11T22:47:31.424934+0000) 2019-12-11T22:48:10.829 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:10.823+0000 1694f700 -1 osd.7 349 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:11.110 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:48:11.104+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 43sec e349 v349) 2019-12-11T22:48:12.233 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:12.047+0000 1694f700 -1 osd.7 350 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:13.613 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:13.543+0000 1694f700 -1 osd.7 351 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112147 2.30s0 2:0c7a7274:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_ec5ecb10b0d74b07b190ab3c11da7e5e:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~128 in=128b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e349) 2019-12-11T22:48:25.924 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:16:12.855 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:25.932 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:16:12.863 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:25.949 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:16:12.880 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:25.950 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:16:12.852 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:25.955 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:16:12.856 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:25.962 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:16:12.893 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:26.182 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:16:13.113 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:26.254 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:16:13.155 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:26.260 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:16:13.191 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:48:41.707 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:48:41.583+0000 21452700 -1 reset not still connected to 0x1c7655e0 2019-12-11T22:48:41.716 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:48:41.583+0000 21452700 -1 reset not still connected to 0x1c7655e0 2019-12-11T22:48:52.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:48:52.035+0000 35b8e700 -1 osd.7 354 build_incremental_map_msg missing incremental map 337 2019-12-11T22:49:10.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:10.548+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:10.983 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:10.978+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:11.214 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:11.207+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:11.391 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:11.385+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:11.552 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:11.547+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:11.967 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:11.962+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:12.223 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:12.215+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:12.358 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:12.353+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:12.566 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:12.562+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:13.008 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:13.003+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:13.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:13.188+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:13.256 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:13.250+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:13.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:13.316+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:13.617 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:13.612+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:14.043 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:14.037+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:14.180 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:14.172+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:14.199 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:14.192+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:14.203 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:14.198+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:14.302 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:14.295+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:14.360 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:14.354+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:14.642 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:14.634+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:15.082 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:15.075+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:15.135 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:15.128+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:15.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:15.216+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:15.226 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:15.221+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:15.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:15.311+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:15.335 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:15.328+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:15.680 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:15.674+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:15.682 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:15.678+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:16.094 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:16.089+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:16.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:16.108+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:16.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:16.186+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:16.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:16.188+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:16.295 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:16.288+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:16.386 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:16.378+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:16.647 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:16.642+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:16.647 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:16.643+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:16.673 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:16.666+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112317 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 821202576 gen 81] snapc 0=[] ondisk+write+known_if_redirected e353) 2019-12-11T22:49:17.117 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:17.111+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:17.117 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:17.111+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:17.199 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:17.194+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:17.203 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:17.198+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:17.278 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:17.272+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:17.415 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:17.408+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:17.670 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:17.664+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:17.670 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:17.665+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:18.076 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:18.070+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:18.162 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:18.156+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:18.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:18.206+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:18.215 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:18.209+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:18.242 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:18.237+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:18.408 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:18.401+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:18.617 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:18.611+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:18.643 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:18.637+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:18.644 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:18.638+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:19.065 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:19.059+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:19.167 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:19.162+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:19.169 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:19.165+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:19.190 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:19.185+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:19.273 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:19.265+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:19.401 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:19.395+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:19.581 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:19.576+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:19.608 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:19.597+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:19.608 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:19.600+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:20.042 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:20.035+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:20.146 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:20.140+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:20.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:20.185+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:20.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:20.188+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:20.245 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:20.240+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:20.355 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:20.350+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:20.553 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:20.548+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:20.591 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:20.586+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:20.593 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:20.590+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:21.068 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:21.060+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:21.112 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:21.105+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:21.162 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:21.156+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:21.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:21.160+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:21.219 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:21.214+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:21.324 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:49:21.319+0000 1694f700 -1 osd.2 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:24.665634+0000 front 2019-12-11T22:48:24.896333+0000 (oldest deadline 2019-12-11T22:49:10.518326+0000) 2019-12-11T22:49:21.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:21.567+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:21.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:21.568+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:21.602 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:21.597+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:22.026 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:22.020+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:22.136 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:22.131+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:22.177 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:22.171+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:22.177 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:22.172+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:22.183 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:22.176+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:22.580 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:22.574+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:22.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:22.577+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:22.625 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:22.619+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:23.020 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:23.014+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:23.157 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:23.152+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:23.160 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:23.156+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:23.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:23.162+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:23.187 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:23.182+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:23.605 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:23.598+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:23.606 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:23.601+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:23.609 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:23.605+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:23.611 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:23.608+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:24.023 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:24.016+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:24.172 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:24.165+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:24.204 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:24.199+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:24.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:24.205+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:24.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:24.209+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:24.596 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:24.588+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:24.597 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:24.591+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:24.632 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:24.627+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:24.633 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:24.629+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:25.026 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:25.018+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:25.164 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:25.159+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:25.199 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:25.192+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:25.225 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:25.218+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:25.227 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:25.222+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:25.638 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:25.630+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:25.639 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:25.633+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:25.652 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:25.647+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:25.653 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:25.649+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:26.045 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:26.037+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:26.193 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:26.178+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:26.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:26.181+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:26.194 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:26.185+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:26.215 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:26.208+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:26.676 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:26.672+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:26.677 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:26.673+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:26.684 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:26.678+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:26.687 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:26.682+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:27.083 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:27.076+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:27.188 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:27.182+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:27.217 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:27.210+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:27.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:27.216+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:27.225 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:27.219+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:27.712 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:27.703+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:27.714 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:27.708+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:27.717 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:27.711+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:27.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:27.714+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:28.123 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:28.116+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:28.142 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:28.137+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:28.205 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:28.198+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:28.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:28.234+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:28.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:28.237+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:28.697 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:28.690+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:28.698 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:28.691+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:28.760 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:28.756+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:28.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:28.758+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:29.093 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:29.082+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:29.163 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:29.159+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:29.217 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:29.209+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:29.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:29.277+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:29.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:29.279+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:29.709 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:29.701+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:29.710 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:29.704+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:29.790 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:29.785+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:29.792 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:29.789+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:30.128 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:30.120+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:30.203 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:30.195+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:30.206 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:30.198+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:30.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:30.241+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:30.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:30.243+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:30.701 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:30.693+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:30.704 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:30.699+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:30.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:30.775+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:30.782 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:30.777+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:31.101 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:31.094+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:31.237 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:31.227+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:31.257 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:31.250+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:31.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:31.263+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:31.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:31.265+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:31.660 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:31.653+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:31.664 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:31.658+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:31.742 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:31.737+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:31.743 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:31.739+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:32.132 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:32.125+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:32.200 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:32.194+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:32.214 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:32.206+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:32.261 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:32.253+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:32.261 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:32.255+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:32.652 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:32.644+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:32.652 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:32.646+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:32.782 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:32.777+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:32.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:32.782+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:33.102 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:33.094+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:33.163 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:33.152+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:33.186 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:33.179+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:33.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:33.219+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:33.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:33.221+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:33.607 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:33.599+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:33.608 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:33.602+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:33.770 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:33.762+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:33.771 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:33.765+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:34.088 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:34.081+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:34.203 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:34.193+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:34.205 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:34.196+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:34.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:34.261+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:34.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:34.266+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:34.602 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:34.590+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:34.604 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:34.593+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:34.742 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:34.737+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:34.743 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:34.738+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:35.074 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:35.066+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:35.174 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:35.170+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:35.177 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:35.169+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:35.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:35.276+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:35.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:35.278+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:35.602 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:35.594+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:35.602 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:35.596+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:35.735 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:35.730+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:35.736 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:35.732+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:36.124 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:36.117+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:36.158 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:36.148+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:36.203 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:36.196+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:36.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:36.307+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:36.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:36.310+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:36.590 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:36.582+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:36.590 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:36.584+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:36.776 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:36.771+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:36.777 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:36.773+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:37.149 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:37.141+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:37.197 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:37.193+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:37.241 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:37.235+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:37.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:37.271+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:37.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:37.274+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:37.581 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:37.573+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:37.582 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:37.576+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:37.736 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:37.731+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:37.736 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:37.733+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:38.148 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:38.140+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:38.225 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:38.220+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:38.242 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:38.234+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:38.243 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:38.237+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:38.297 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:38.289+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:38.538 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:38.530+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:38.539 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:38.533+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:38.774 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:38.770+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:38.777 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:38.773+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:39.127 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:39.119+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:39.184 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:39.178+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:39.236 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:39.228+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:39.237 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:39.231+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:39.286 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:39.278+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:39.589 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:39.583+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:39.593 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:39.587+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:39.749 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:39.742+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:39.750 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:39.745+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:40.148 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:40.141+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:40.156 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:40.151+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:40.218 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:40.208+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:40.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:40.212+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:40.295 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:40.288+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:40.564 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:40.556+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:40.565 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:40.559+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:40.772 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:40.767+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:40.773 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:40.769+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:41.134 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:41.129+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:41.166 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:41.159+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:41.212 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:41.203+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:41.212 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:41.206+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:41.302 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:41.295+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:41.583 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:41.575+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:41.584 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:41.578+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:41.632 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:49:41.628+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e354 v354) 2019-12-11T22:49:41.760 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:41.754+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:41.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:41.757+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:42.094 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:42.090+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:42.186 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:42.178+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:42.186 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:42.181+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:42.213 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:42.206+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:42.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:42.343+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:42.563 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:42.556+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:42.567 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:42.561+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:42.807 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:42.801+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:42.807 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:42.803+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:43.061 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:43.056+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:43.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:43.205+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:43.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:43.208+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:43.238 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:43.231+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:43.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:43.307+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:43.533 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:43.527+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:43.535 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:43.530+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:43.853 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:43.848+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:43.854 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:43.850+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:44.057 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:44.052+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:44.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:44.180+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:44.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:44.183+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:44.214 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:44.207+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:44.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:44.296+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:44.568 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:44.560+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:44.573 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:44.565+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:44.849 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:44.842+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:44.849 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:44.846+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:45.012 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:45.007+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:45.200 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:45.193+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:45.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:45.209+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:45.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:45.217+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:45.275 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:45.267+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:45.611 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:45.603+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:45.611 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:45.606+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:45.810 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:45.804+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:45.810 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:45.806+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:46.053 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:46.048+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:46.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:46.188+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:46.216 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:46.209+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:46.217 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:46.211+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:46.295 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:46.288+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:46.603 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:46.595+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:46.604 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:46.597+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:46.729 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:49:46.722+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e354 v354) 2019-12-11T22:49:46.811 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:17:33.742 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:46.818 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:17:33.750 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:46.827 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:17:33.757 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:46.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:46.842+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:46.849 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:46.845+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:46.872 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:17:33.803 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:46.894 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:17:33.795 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:46.903 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:17:33.804 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:46.934 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:17:33.865 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:47.004 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:17:33.934 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:47.056 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:47.052+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:47.116 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:17:34.018 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:49:47.238 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:47.232+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:47.247 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:47.239+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:47.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:47.242+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:47.288 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:47.281+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:47.611 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:47.604+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:47.612 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:47.606+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:47.830 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:47.824+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:47.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:47.826+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:48.090 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:48.085+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:48.215 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:48.208+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:48.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:48.248+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:48.257 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:48.251+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:48.269 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:48.262+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:48.583 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:48.576+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:48.584 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:48.578+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:48.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:48.834+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:48.845 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:48.837+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:49.141 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:49.136+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:49.237 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:49.229+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:49.258 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:49.251+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:49.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:49.291+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:49.300 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:49.294+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:49.617 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:49.610+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:49.620 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:49.614+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:49.815 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:49.809+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:49.819 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:49.814+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:50.115 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:50.110+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:50.270 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:50.262+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:50.287 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:50.279+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:50.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:50.282+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:50.292 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:50.287+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:50.607 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:50.599+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:50.608 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:50.601+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:50.809 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:50.802+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:50.809 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:50.805+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:51.142 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:49:51.138+0000 1694f700 -1 osd.0 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.361735+0000 front 2019-12-11T22:48:27.358448+0000 (oldest deadline 2019-12-11T22:49:10.773164+0000) 2019-12-11T22:49:51.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:51.274+0000 1694f700 -1 osd.7 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.516589+0000 front 2019-12-11T22:48:27.478795+0000 (oldest deadline 2019-12-11T22:49:13.348648+0000) 2019-12-11T22:49:51.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:49:51.275+0000 1694f700 -1 osd.7 354 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2.d5ffb260 (undecoded) ondisk+write+known_if_redirected e352) 2019-12-11T22:49:51.298 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:49:51.292+0000 1694f700 -1 osd.4 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.833742+0000 front 2019-12-11T22:48:26.037208+0000 (oldest deadline 2019-12-11T22:49:10.480485+0000) 2019-12-11T22:49:51.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:49:51.307+0000 1694f700 -1 osd.6 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:27.192263+0000 front 2019-12-11T22:48:27.444258+0000 (oldest deadline 2019-12-11T22:49:12.983734+0000) 2019-12-11T22:49:51.599 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:51.591+0000 1694f700 -1 osd.5 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:48:43.390148+0000 (oldest deadline 2019-12-11T22:49:23.390148+0000) 2019-12-11T22:49:51.600 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:49:51.594+0000 1694f700 -1 osd.5 354 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e353) 2019-12-11T22:49:51.796 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:49:51.788+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e354 v354) 2019-12-11T22:49:51.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:51.836+0000 1694f700 -1 osd.3 354 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:48:25.555123+0000 front 2019-12-11T22:48:25.673308+0000 (oldest deadline 2019-12-11T22:49:10.154309+0000) 2019-12-11T22:49:51.843 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:51.839+0000 1694f700 -1 osd.3 354 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:52.888 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:49:52.884+0000 1694f700 -1 osd.3 355 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:112312 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 384106720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e352) 2019-12-11T22:49:54.126 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:49:54.122+0000 20e25700 -1 RGWWatcher::handle_error cookie 384106720 err (107) Transport endpoint is not connected 2019-12-11T22:50:24.752 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:50:24.745+0000 35f8f700 -1 osd.1 360 build_incremental_map_msg missing incremental map 360 2019-12-11T22:50:24.771 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:50:24.758+0000 34f8d700 -1 osd.1 360 build_incremental_map_msg missing incremental map 360 2019-12-11T22:50:24.783 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:50:24.771+0000 35f8f700 -1 osd.1 360 build_incremental_map_msg missing incremental map 360 2019-12-11T22:50:24.936 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:50:24.923+0000 32788700 -1 osd.1 360 build_incremental_map_msg missing incremental map 360 2019-12-11T22:50:25.002 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:50:24.981+0000 31786700 -1 osd.1 360 build_incremental_map_msg missing incremental map 360 2019-12-11T22:50:25.017 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:50:24.984+0000 31786700 -1 osd.1 360 build_incremental_map_msg missing incremental map 360 2019-12-11T22:50:52.990 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:50:52.984+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:50:54.032 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:50:54.026+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:50:55.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:50:55.053+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:50:56.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:50:56.069+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:50:57.111 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:50:57.105+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:50:58.146 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:50:58.140+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:50:59.124 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:50:59.117+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:00.154 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:00.148+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:01.180 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:01.174+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:02.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:02.196+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:03.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:03.172+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:04.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:04.223+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:05.220 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:05.215+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:06.246 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:06.240+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:07.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:07.292+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:08.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:08.261+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:09.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:09.261+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:10.297 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:10.291+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:11.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:11.271+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:12.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:12.293+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:13.309 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:13.304+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:14.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:14.308+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:15.328 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:15.323+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:16.361 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:16.357+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:17.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:17.339+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:18.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:18.360+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:19.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:19.364+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:20.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:20.383+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:21.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:21.344+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:22.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:22.320+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:23.296 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:23.291+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:23.310 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:23.305+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:23.343 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:23.336+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:24.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:24.342+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:24.356 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:24.348+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:24.393 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:24.386+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:24.689 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:24.684+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:25.023 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:25.017+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:25.308 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:25.303+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:25.360 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:25.355+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:25.361 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:25.355+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:25.411 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:25.405+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:25.728 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:25.722+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:26.007 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:26.001+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:26.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:26.273+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:26.350 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:26.345+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:26.385 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:26.380+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:26.393 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:26.385+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:26.693 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:26.687+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:27.029 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:27.016+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:27.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:27.262+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:27.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:27.263+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:27.349 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:27.343+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:27.398 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:27.388+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:27.438 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:27.433+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:27.678 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:27.671+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:28.023 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:28.017+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:28.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:28.244+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:28.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:28.245+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:28.356 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:28.350+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:28.378 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:28.372+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:28.419 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:28.409+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:28.698 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:28.693+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:28.980 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:28.973+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:29.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:29.291+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:29.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:29.293+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:29.344 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:29.337+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:29.393 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:29.387+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:29.447 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:29.441+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:29.725 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:29.719+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:30.009 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:30.003+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:30.302 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:30.297+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:30.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:30.331+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:30.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:30.333+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:30.389 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:30.383+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:30.442 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:30.437+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:30.745 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:30.739+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:31.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:31.030+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:31.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:31.303+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:31.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:31.310+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:31.317 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:31.307+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:31.392 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:31.387+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:31.491 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:31.486+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:31.736 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:31.731+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:32.023 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:32.017+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:32.285 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:32.280+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:32.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:32.317+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:32.325 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:32.318+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:32.372 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:32.368+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:32.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:32.444+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:32.714 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:32.709+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:33.066 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:33.059+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:33.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:33.289+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:33.297 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:33.292+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:33.304 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:33.299+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:33.366 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:33.361+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:33.409 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:33.398+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:33.692 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:33.687+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:34.064 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:34.057+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:34.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:34.266+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:34.274 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:34.269+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:34.350 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:34.345+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:34.400 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:34.395+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:34.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:34.420+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:34.693 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:34.688+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:35.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:35.082+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:35.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:35.265+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:35.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:35.266+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:35.393 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:35.389+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:35.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:35.394+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:35.401 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:35.395+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:35.657 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:35.652+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:36.072 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:36.063+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:36.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:36.295+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:36.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:36.297+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:36.390 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:36.383+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:36.392 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:36.388+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:36.438 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:36.432+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:36.710 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:36.705+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:37.043 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:37.036+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:37.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:37.271+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:37.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:37.272+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:37.376 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:37.371+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:37.409 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:37.403+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:37.505 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:37.501+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:37.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:37.748+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:38.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:38.082+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:38.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:38.266+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:38.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:38.267+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:38.345 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:38.341+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:38.402 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:38.396+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:38.558 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:38.553+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:38.763 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:38.759+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:39.046 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:39.040+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:39.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:39.241+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:39.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:39.242+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:39.369 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:39.364+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:39.447 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:39.440+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:39.580 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:39.574+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:39.787 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:39.783+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:40.070 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:40.064+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:40.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:40.243+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:40.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:40.245+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:40.407 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:40.402+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:40.439 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:40.433+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:40.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:40.583+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:40.833 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:40.829+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:41.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:41.057+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:41.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:41.215+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:41.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:41.218+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:41.434 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:41.428+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:41.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:41.452+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:41.558 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:41.553+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:41.880 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:41.876+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:42.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:42.071+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:42.233 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:42.223+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:42.233 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:42.226+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:42.441 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:42.434+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:42.460 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:42.456+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:42.520 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:42.515+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:42.891 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:42.873+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:43.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:43.081+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:43.244 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:43.232+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:43.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:43.238+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:43.464 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:43.459+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:43.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:43.475+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:43.485 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:43.478+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:43.893 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:43.888+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:44.071 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:44.065+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:44.209 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:44.203+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:44.211 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:44.206+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:44.464 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:44.460+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:44.465 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:44.459+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:44.473 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:44.468+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:44.858 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:44.853+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:45.087 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:45.081+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:45.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:45.214+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:45.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:45.215+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:45.442 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:45.435+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:45.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:45.450+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:45.521 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:45.510+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:45.843 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:45.839+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:46.048 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:46.042+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:46.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:46.263+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:46.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:46.267+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:46.444 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:46.438+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:46.502 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:46.496+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:46.560 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:46.551+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:46.886 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:46.881+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:47.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:47.031+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:47.226 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:47.221+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:47.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:47.223+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:47.412 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:47.406+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:47.481 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:47.477+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:47.516 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:47.512+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:47.924 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:47.919+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:48.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:48.056+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:48.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:48.217+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:48.225 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:48.218+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:48.387 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:48.380+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:48.496 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:48.491+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:48.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:48.503+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:48.892 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:48.887+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:49.101 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:49.091+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:49.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:49.260+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:49.267 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:49.261+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:49.422 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:49.415+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:49.507 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:49.502+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:49.514 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:49.508+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:49.934 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:49.930+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:50.069 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:50.061+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:50.241 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:50.235+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:50.241 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:50.236+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:50.413 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:50.400+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:50.505 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:50.498+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:50.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:50.510+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:50.897 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:50.893+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:51.111 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:51.104+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:51.284 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:51.278+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:51.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:51.281+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:51.402 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:51.392+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:51.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:51.534+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:51.540 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:51.534+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:51.893 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:51.889+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:52.114 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:52.108+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:52.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:52.251+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:52.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:52.252+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:52.394 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:52.385+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:52.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:52.512+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:52.583 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:52.578+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:52.876 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:52.872+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:53.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:53.162+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:53.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:53.277+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:53.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:53.278+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:53.377 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:53.372+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:53.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:53.502+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:53.577 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:53.572+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:53.840 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:53.836+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:54.172 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:54.164+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:54.261 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:54.252+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:54.261 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:54.253+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:54.401 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:54.395+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:54.477 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:54.471+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:54.605 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:54.600+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:54.835 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:54.830+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:55.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:55.119+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:55.226 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:55.208+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:55.227 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:55.211+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:55.423 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:55.416+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:55.460 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:55.455+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:55.629 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:55.625+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:55.816 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:55.811+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:56.135 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:56.129+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:56.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:56.262+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:56.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:56.266+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:56.463 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:56.456+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:56.472 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:56.467+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:56.673 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:56.667+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:56.865 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:56.860+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:57.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:57.119+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:57.147 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:51:57.141+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 43sec e360 v360) 2019-12-11T22:51:57.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:57.265+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:57.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:57.266+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:57.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:57.463+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:57.505 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:57.499+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:57.702 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:57.698+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:57.853 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:57.849+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:58.126 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:58.119+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:58.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:58.258+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:58.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:58.259+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:58.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:58.476+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:58.533 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:58.527+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:58.752 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:58.745+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:51:58.828 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:58.823+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:59.089 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:51:59.083+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:51:59.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:59.309+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:51:59.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:51:59.310+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:51:59.513 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:51:59.502+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:51:59.513 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:51:59.499+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:51:59.794 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:51:59.789+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:51:59.796 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:51:59.792+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:52:00.099 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:52:00.091+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:52:00.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:00.314+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:52:00.322 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:00.317+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:00.490 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:52:00.485+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:52:00.515 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:52:00.509+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:52:00.781 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:52:00.776+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:52:00.797 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:52:00.793+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:52:01.081 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:52:01.073+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:52:01.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:01.308+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:52:01.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:01.309+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:01.470 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:52:01.465+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:52:01.508 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:52:01.501+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.294688+0000 front 2019-12-11T22:50:41.149022+0000 (oldest deadline 2019-12-11T22:51:22.447977+0000) 2019-12-11T22:52:01.680 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:52:01.675+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772383+0000 front 2019-12-11T22:50:41.290270+0000 (oldest deadline 2019-12-11T22:52:01.102181+0000) 2019-12-11T22:52:01.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:52:01.744+0000 1694f700 -1 osd.1 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772199+0000 front 2019-12-11T22:50:40.772743+0000 (oldest deadline 2019-12-11T22:51:24.166623+0000) 2019-12-11T22:52:01.789 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:52:01.782+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.159405+0000 front 2019-12-11T22:50:40.174532+0000 (oldest deadline 2019-12-11T22:51:24.463020+0000) 2019-12-11T22:52:02.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:52:02.118+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:41.991563+0000 front 2019-12-11T22:50:42.006406+0000 (oldest deadline 2019-12-11T22:51:24.579152+0000) 2019-12-11T22:52:02.285 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:02.278+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:42.456023+0000 front 2019-12-11T22:50:41.327860+0000 (oldest deadline 2019-12-11T22:51:27.191859+0000) 2019-12-11T22:52:02.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:02.282+0000 1694f700 -1 osd.7 360 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:02.412 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:52:02.409+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 43sec e360 v360) 2019-12-11T22:52:02.458 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:52:02.454+0000 1694f700 -1 osd.3 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:39.037730+0000 front 2019-12-11T22:50:38.906089+0000 (oldest deadline 2019-12-11T22:51:23.235459+0000) 2019-12-11T22:52:02.590 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:19:49.521 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.596 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:19:49.527 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.638 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:19:49.539 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.656 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:19:49.587 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.665 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:19:49.566 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.682 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:52:02.675+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:50:40.772383+0000 front 2019-12-11T22:50:41.290270+0000 (oldest deadline 2019-12-11T22:52:01.102181+0000) 2019-12-11T22:52:02.698 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:19:49.629 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.852 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:19:49.751 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.903 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:19:49.834 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:02.911 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:19:49.842 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:03.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:03.270+0000 1694f700 -1 osd.7 361 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:04.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:04.373+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:05.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:05.375+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:06.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:06.423+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:07.461 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:07.455+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:08.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:08.421+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:09.402 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:09.397+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:10.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:10.425+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:11.434 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:11.429+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:12.464 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:12.459+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:13.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:13.467+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:14.042 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:20:00.973 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.049 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:20:00.980 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.063 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:20:00.994 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.065 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:20:00.966 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.069 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:20:00.970 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.070 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:20:01.001 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.098 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:20:01.023 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.109 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:20:01.040 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.118 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:20:01.020 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:52:14.425 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:14.421+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:15.404 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:15.399+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:16.451 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:16.447+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:17.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:17.425+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:18.421 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:18.415+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:19.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:19.382+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:20.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:20.398+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:21.412 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:21.406+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:22.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:22.380+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:23.379 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:23.369+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:24.423 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:24.418+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:25.417 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:25.411+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:26.443 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:26.437+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:27.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:27.438+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:28.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:28.468+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:29.456 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:29.448+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:30.460 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:30.454+0000 1694f700 -1 osd.7 362 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:31.499 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:31.494+0000 1694f700 -1 osd.7 363 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:32.547 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:32.542+0000 1694f700 -1 osd.7 363 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:33.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:33.861+0000 1694f700 -1 osd.7 364 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:34.986 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:34.971+0000 1694f700 -1 osd.7 365 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:36.025 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:36.018+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:36.990 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:36.986+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:37.983 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:37.978+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:38.991 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:38.985+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:39.999 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:39.994+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:41.046 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:41.040+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:42.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:42.064+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:43.096 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:43.090+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:44.098 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:44.093+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:45.124 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:45.118+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:46.131 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:46.123+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:47.160 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:47.152+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:48.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:48.179+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:49.227 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:49.221+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:50.226 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:50.221+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:51.200 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:51.195+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:52.187 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:52.181+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:53.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:53.177+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:54.174 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:54.167+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:55.154 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:55.148+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:56.142 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:56.136+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:57.155 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:57.150+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:58.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:58.111+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:52:59.093 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:52:59.087+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:00.067 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:00.061+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:01.100 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:01.093+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:01.862 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:01.856+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:02.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:02.065+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:02.145 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:02.137+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:02.851 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:02.845+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:02.859 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:02.854+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:03.050 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:03.044+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:03.191 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:03.185+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:03.809 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:03.804+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:03.842 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:03.837+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:03.953 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:03.945+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:04.033 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:04.027+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:04.223 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:04.219+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:04.803 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:04.795+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:04.835 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:04.830+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:04.927 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:04.920+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:05.023 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:05.017+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:05.033 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:05.026+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:05.065 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:05.058+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:05.236 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:05.230+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:05.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:05.788+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:05.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:05.826+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:05.890 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:05.883+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:06.025 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:06.017+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:06.040 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:06.034+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:06.068 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:06.061+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:06.231 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:06.224+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:06.783 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:06.778+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:06.784 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:06.779+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:06.946 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:06.940+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:06.996 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:06.989+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:07.052 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:07.044+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:07.079 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:07.073+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:07.197 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:07.192+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:07.741 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:07.736+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:07.757 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:07.753+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:07.758 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:07.755+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:07.983 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:07.976+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:08.001 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:07.994+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:08.057 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:08.051+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:08.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:08.112+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:08.204 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:08.198+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:08.703 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:08.699+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:08.778 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:08.773+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:08.779 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:08.774+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:09.003 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:08.995+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:09.012 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:08.999+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:09.034 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:09.029+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:09.088 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:09.083+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:09.222 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:09.217+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:09.678 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:09.674+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:09.759 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:09.753+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:09.759 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:09.755+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:09.967 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:09.961+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:10.010 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:10.001+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:10.020 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:10.013+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:10.088 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:10.083+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:10.225 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:10.219+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:10.656 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:10.650+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:10.725 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:10.720+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:10.726 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:10.722+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:10.998 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:10.991+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:11.001 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:10.996+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:11.011 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:11.004+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:11.028 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:11.022+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:11.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:11.114+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:11.239 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:11.235+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:11.643 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:11.639+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:11.682 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:11.678+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:11.683 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:11.680+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:11.982 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:11.973+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:11.982 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:11.977+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:12.026 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:12.019+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:12.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:12.029+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:12.130 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:12.126+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:12.212 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:12.207+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:12.654 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:12.650+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:12.713 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:12.706+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:12.713 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:12.707+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:12.997 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:12.990+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:12.998 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:12.993+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:13.006 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:13.000+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:13.027 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:13.021+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:13.180 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:13.176+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:13.226 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:13.221+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:13.684 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:13.679+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:13.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:13.756+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:13.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:13.757+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:14.012 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:14.005+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:14.034 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:14.026+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:14.034 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:14.029+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:14.080 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:14.073+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:14.157 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:14.153+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:14.231 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:14.227+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:14.718 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:14.712+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:14.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:14.714+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:14.724 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:14.719+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:15.062 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:15.056+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:15.081 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:15.060+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:15.081 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:15.063+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:15.111 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:15.105+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:15.145 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:15.141+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:15.274 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:15.269+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:15.743 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:15.739+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:15.745 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:15.740+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:15.757 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:15.751+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:16.073 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:16.066+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:16.075 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:16.068+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:16.115 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:16.106+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:16.116 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:16.109+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:16.130 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:16.124+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:16.295 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:16.290+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:16.697 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:16.692+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:16.700 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:16.696+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:16.761 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:16.755+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:17.092 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:17.085+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:17.120 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:17.113+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:17.129 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:17.122+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:17.130 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:17.123+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:17.133 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:17.127+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:17.291 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:17.286+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:17.684 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:17.679+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:17.685 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:17.681+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:17.734 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:17.730+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:18.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:18.059+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:18.124 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:18.119+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:18.140 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:18.132+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:18.185 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:18.175+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:18.186 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:18.179+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:18.260 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:18.256+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:18.660 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:18.653+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:18.660 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:18.655+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:18.780 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:18.775+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:19.035 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:19.027+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:19.109 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:19.103+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:19.161 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:19.152+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:19.162 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:19.155+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:19.190 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:19.182+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:19.254 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:19.248+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:19.614 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:19.609+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:19.616 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:19.613+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:19.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:19.750+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:20.057 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:20.049+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:20.127 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:20.121+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:20.167 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:20.160+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:20.186 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:20.178+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:20.187 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:20.181+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:20.292 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:20.289+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:20.587 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:20.583+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:20.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:20.584+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:20.763 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:20.758+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:21.101 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:21.094+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:21.133 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:21.128+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:21.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:21.175+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:21.207 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:21.198+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:21.208 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:21.201+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:21.319 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:21.314+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:21.640 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:21.634+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:21.640 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:21.636+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:21.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:21.789+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:22.076 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:22.068+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:22.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:22.170+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:22.215 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:22.206+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:22.230 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:22.221+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:22.231 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:22.224+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:22.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:22.291+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:22.645 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:22.640+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:22.647 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:22.643+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:22.835 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:22.830+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:23.056 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:23.050+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:23.182 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:23.176+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:23.183 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:23.178+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:23.205 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:23.200+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:23.259 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:23.252+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:23.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:23.273+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:23.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:23.647+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:23.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:23.650+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:23.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:23.787+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:24.045 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:24.038+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:24.192 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:24.182+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:24.196 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:24.189+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:24.218 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:24.213+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:24.231 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:24.224+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:24.267 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:24.263+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:24.680 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:24.675+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:24.681 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:24.677+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:24.757 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:24.752+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:25.075 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:25.066+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:25.149 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:25.142+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:25.150 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:25.144+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:25.194 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:25.188+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:25.199 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:25.194+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:25.227 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:25.222+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:25.687 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:25.682+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:25.689 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:25.686+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:25.722 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:25.717+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:26.122 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:26.116+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:26.164 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:26.158+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:26.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:26.160+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:26.167 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:26.162+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:26.227 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:26.216+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:26.249 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:26.242+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:26.701 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:26.696+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:26.702 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:26.699+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:26.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:26.753+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:27.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:27.102+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:27.193 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:27.185+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:27.193 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:27.188+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:27.197 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:27.191+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:27.203 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:27.198+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:27.245 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:27.239+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:27.687 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:27.682+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:27.690 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:27.686+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:27.719 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:27.712+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:28.133 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:28.124+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:28.184 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:28.175+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:28.186 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:28.180+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:28.234 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:28.225+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:28.237 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:28.232+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:28.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:28.245+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:28.662 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:28.658+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:28.665 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:28.662+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:28.722 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:28.716+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:29.105 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:29.099+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:29.148 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:29.142+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:29.152 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:29.146+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:29.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:29.241+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:29.266 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:29.259+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:29.287 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:29.277+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:29.720 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:29.712+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:29.721 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:29.714+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:29.726 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:29.720+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:30.128 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:30.122+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:30.131 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:30.126+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:30.148 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:30.141+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:30.271 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:30.264+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:30.285 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:30.279+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:30.298 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:30.293+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:30.711 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:30.706+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:30.757 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:30.749+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:30.757 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:30.752+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:31.120 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:31.113+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:31.123 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:31.117+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:31.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:31.165+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:31.238 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:31.230+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:31.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:31.249+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:31.259 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:31.253+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:31.339 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:31.335+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:31.724 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:31.720+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:31.793 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:31.789+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:31.795 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:31.792+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:32.133 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:32.126+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:32.141 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:32.133+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:32.142 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:32.136+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:32.216 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:32.209+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:32.284 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:32.276+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:32.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:32.282+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:32.329 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:32.325+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:32.684 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:32.674+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:32.777 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:32.773+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:32.778 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:32.774+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:33.142 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:33.135+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:33.156 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:33.148+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:33.157 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:33.151+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:33.246 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:33.239+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:33.284 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:33.280+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:33.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:33.307+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:33.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:33.310+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:33.642 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:33.636+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:33.795 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:33.789+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:33.796 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:33.791+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:34.129 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:34.121+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:34.130 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:34.124+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:34.174 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:34.167+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:34.212 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:34.205+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:34.283 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:34.278+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:34.343 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:34.335+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:34.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:34.338+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:34.644 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:34.639+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:34.767 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:34.762+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:34.767 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:34.764+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:35.130 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:35.120+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:35.130 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:35.124+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:35.171 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:35.164+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:35.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:35.188+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:35.286 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:35.282+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:35.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:35.343+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:35.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:35.345+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:35.667 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:35.662+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:35.760 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:35.752+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:35.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:35.757+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:36.123 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:36.117+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:36.126 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:36.121+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:36.164 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:36.158+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:36.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:36.170+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:36.275 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:36.270+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:36.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:36.317+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:36.325 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:36.319+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:36.676 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:36.669+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:36.745 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:36.739+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:36.746 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:36.742+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:37.093 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:37.086+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:37.096 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:37.091+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:37.179 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:37.171+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:37.189 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:37.182+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:37.270 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:37.266+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:37.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:37.282+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:37.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:37.284+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:37.679 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:53:37.673+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e366 v366) 2019-12-11T22:53:37.680 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:37.674+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:37.754 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:37.747+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:37.754 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:37.749+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:38.132 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:38.124+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:38.133 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:38.126+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:38.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:38.188+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:38.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:38.227+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:38.241 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:38.232+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:38.244 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:38.236+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:38.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:38.239+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:38.729 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:38.724+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:38.795 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:38.789+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:38.795 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:38.790+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:39.160 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:39.153+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:39.160 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:39.155+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:39.164 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:39.158+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:39.201 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:39.195+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:39.237 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:39.229+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:39.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:39.232+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:39.255 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:39.248+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:39.691 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:39.683+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:39.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:39.778+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:39.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:39.780+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:40.149 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:40.142+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:40.152 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:40.144+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:40.188 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:40.183+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:40.189 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:40.180+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:40.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:40.267+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:40.274 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:40.268+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:40.276 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:40.270+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:40.734 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:40.728+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:40.808 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:40.802+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:40.810 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:40.806+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:41.160 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:41.154+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:41.178 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:41.171+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:41.182 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:41.176+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:41.234 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:41.221+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:41.251 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:41.242+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:41.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:41.282+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:41.292 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:41.284+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:41.734 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:41.728+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:41.813 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:41.806+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:41.814 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:41.808+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:42.134 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:42.128+0000 1694f700 -1 osd.5 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 ever on either front or back, first ping sent 2019-12-11T22:52:30.935870+0000 (oldest deadline 2019-12-11T22:53:10.935870+0000) 2019-12-11T22:53:42.135 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:42.129+0000 1694f700 -1 osd.5 366 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:42.175 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:53:42.169+0000 1694f700 -1 osd.2 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.479923+0000 front 2019-12-11T22:52:18.540623+0000 (oldest deadline 2019-12-11T22:53:01.847694+0000) 2019-12-11T22:53:42.236 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:53:42.229+0000 1694f700 -1 osd.6 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.892971+0000 front 2019-12-11T22:52:18.706894+0000 (oldest deadline 2019-12-11T22:53:03.394301+0000) 2019-12-11T22:53:42.281 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:53:42.273+0000 1694f700 -1 osd.4 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:20.269556+0000 front 2019-12-11T22:52:19.916136+0000 (oldest deadline 2019-12-11T22:53:04.846910+0000) 2019-12-11T22:53:42.328 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:42.321+0000 1694f700 -1 osd.7 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:19.322526+0000 front 2019-12-11T22:52:19.117543+0000 (oldest deadline 2019-12-11T22:53:30.678399+0000) 2019-12-11T22:53:42.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:53:42.323+0000 1694f700 -1 osd.7 366 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112308 2.20s0 2:064dffab:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_f1b603194e9e4128a249788a11b69d80:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~142 in=142b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e358) 2019-12-11T22:53:42.739 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:53:42.733+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 44sec e366 v366) 2019-12-11T22:53:42.746 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:53:42.741+0000 1694f700 -1 osd.0 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:17.662956+0000 front 2019-12-11T22:52:17.537490+0000 (oldest deadline 2019-12-11T22:53:01.608755+0000) 2019-12-11T22:53:42.773 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:42.768+0000 1694f700 -1 osd.3 366 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:52:18.627636+0000 front 2019-12-11T22:52:18.622679+0000 (oldest deadline 2019-12-11T22:53:02.496576+0000) 2019-12-11T22:53:42.775 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:42.771+0000 1694f700 -1 osd.3 366 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:43.202 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:53:43.155+0000 1694f700 -1 osd.5 367 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88519 7.0 7.cc36bc80 (undecoded) ondisk+retry+read+known_if_redirected e364) 2019-12-11T22:53:43.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:53:43.776+0000 1694f700 -1 osd.3 367 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:112691 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 822028720 gen 1] snapc 0=[] ondisk+write+known_if_redirected e366) 2019-12-11T22:53:44.984 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:53:44.963+0000 20e25700 -1 RGWWatcher::handle_error cookie 822028720 err (107) Transport endpoint is not connected 2019-12-11T22:54:43.810 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:54:43.805+0000 35f8f700 -1 osd.1 371 build_incremental_map_msg missing incremental map 338 2019-12-11T22:54:43.819 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:54:43.815+0000 35f8f700 -1 osd.1 371 build_incremental_map_msg missing incremental map 349 2019-12-11T22:54:43.827 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:54:43.823+0000 35f8f700 -1 osd.1 371 build_incremental_map_msg missing incremental map 360 2019-12-11T22:54:44.677 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:54:44.622+0000 35f8f700 -1 osd.5 371 build_incremental_map_msg missing incremental map 337 2019-12-11T22:54:49.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:54:49.464+0000 35f8f700 -1 osd.3 371 build_incremental_map_msg missing incremental map 337 2019-12-11T22:55:13.497 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:13.492+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:13.656 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:13.651+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:14.431 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:14.425+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:14.470 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:14.466+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:14.625 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:14.620+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:15.446 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:15.442+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:15.448 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:15.441+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:15.597 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:15.592+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:16.445 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:16.441+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:16.457 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:16.450+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:16.635 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:16.630+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:16.769 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:16.762+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:16.770 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:16.763+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:17.263 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:17.251+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:17.432 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:17.427+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:17.477 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:17.470+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:17.592 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:17.587+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:17.758 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:17.751+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:17.787 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:17.782+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:18.230 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:18.226+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:18.473 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:18.468+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:18.475 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:18.469+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:18.561 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:18.556+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:18.785 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:18.778+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:18.839 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:18.831+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:18.863 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:18.858+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:19.278 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:19.268+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:19.442 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:19.437+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:19.457 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:19.451+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:19.605 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:19.600+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:19.808 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:19.802+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:19.845 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:19.842+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:19.882 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:19.876+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:20.256 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:20.250+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:20.443 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:20.439+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:20.490 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:20.483+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:20.585 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:20.579+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:20.795 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:20.789+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:20.839 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:20.834+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:20.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:20.844+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:21.272 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:21.268+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:21.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:21.473+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:21.515 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:21.509+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:21.537 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:21.532+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:21.787 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:21.780+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:21.884 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:21.880+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:21.890 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:21.884+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:22.286 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:22.281+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:22.469 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:22.463+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:22.509 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:22.504+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:22.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:22.583+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:22.764 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:22.756+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:22.837 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:22.833+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:22.921 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:22.915+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:23.283 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:23.278+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:23.432 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:23.425+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:23.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:23.457+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:23.550 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:23.544+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:23.759 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:23.753+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:23.821 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:23.815+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:23.967 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:23.960+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:24.311 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:24.307+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:24.395 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:24.390+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:24.424 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:24.421+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:24.561 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:24.557+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:24.736 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:24.730+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:24.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:24.773+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:24.963 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:24.955+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:25.361 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:25.355+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:25.385 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:25.379+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:25.419 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:25.416+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:25.571 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:25.567+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:25.751 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:25.744+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:25.791 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:25.786+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:25.969 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:25.959+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:26.359 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:26.354+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:26.390 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:26.384+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:26.465 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:26.461+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:26.563 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:26.558+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:26.755 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:26.748+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:26.776 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:26.767+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:26.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:26.985+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:27.381 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:27.376+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:27.440 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:27.433+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:27.456 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:27.451+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:27.527 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:27.522+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:27.751 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:27.744+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:27.809 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:27.804+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:27.975 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:27.968+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:28.388 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:28.384+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:28.468 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:28.461+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:28.472 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:28.468+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:28.540 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:28.535+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:28.755 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:28.748+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:28.828 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:28.823+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:28.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:28.924+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:29.353 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:29.348+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:29.434 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:29.428+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:29.483 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:29.477+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:29.552 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:29.547+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:29.802 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:29.796+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:29.817 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:29.812+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:29.964 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:29.957+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:30.320 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:30.316+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:30.438 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:30.432+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:30.523 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:30.517+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:30.560 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:30.553+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:30.783 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:30.778+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:30.818 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:30.795+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:30.981 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:30.976+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:31.302 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:31.298+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:31.489 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:31.482+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:31.506 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:31.501+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:31.589 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:31.585+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:31.769 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:31.764+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:31.816 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:31.809+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:31.936 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:31.929+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:32.287 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:32.282+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:32.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:32.472+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:32.488 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:32.482+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:32.561 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:32.547+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:32.759 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:32.746+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:32.808 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:32.798+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:32.954 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:32.948+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:33.325 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:33.312+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:33.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:33.443+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:33.498 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:33.482+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:33.533 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:33.510+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:33.789 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:33.783+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:33.793 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:33.788+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:33.969 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:33.963+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:34.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:34.325+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:34.440 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:34.436+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:34.444 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:34.436+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:34.562 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:34.557+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:34.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:34.773+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:34.805 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:34.798+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:35.007 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:35.002+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:35.308 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:35.303+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:35.432 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:35.426+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:35.475 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:35.469+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:35.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:35.512+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:35.760 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:35.756+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:35.823 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:35.818+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:35.987 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:35.981+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:36.314 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:36.310+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:36.438 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:36.432+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:36.441 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:36.435+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:36.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:36.512+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:36.744 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:36.739+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:36.801 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:36.795+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:36.965 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:36.956+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:36.965 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:36.957+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:37.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:37.331+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:37.447 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:37.442+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:37.477 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:37.473+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:37.480 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:37.473+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:37.777 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:37.772+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:37.811 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:37.804+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:37.960 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:37.954+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:37.961 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:37.955+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:38.334 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:38.330+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:38.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:38.458+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:38.486 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:38.482+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:38.523 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:38.518+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:38.750 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:38.745+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:38.830 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:38.823+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:38.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:38.927+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:38.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:38.928+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:39.377 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:39.372+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:39.448 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:39.440+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:39.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:39.474+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:39.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:39.528+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:39.780 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:55:39.775+0000 1694f700 -1 osd.1 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:32.813781+0000 front 2019-12-11T22:54:32.840859+0000 (oldest deadline 2019-12-11T22:55:18.075350+0000) 2019-12-11T22:55:39.831 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:39.824+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:39.980 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:39.973+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:39.982 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:39.977+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:40.335 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:40.329+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:40.408 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:40.403+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:40.497 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:40.492+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:40.514 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:40.509+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:40.831 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:40.824+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:41.011 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:41.004+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:41.012 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:41.006+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:41.342 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:41.336+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:41.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:41.445+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:41.512 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:41.507+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:41.558 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:41.541+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:41.786 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:41.779+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:41.974 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:41.967+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:41.974 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:41.969+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:42.344 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:42.339+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:42.409 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:42.403+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:42.488 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:42.482+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:42.513 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:42.507+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:42.772 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:42.762+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:43.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:42.995+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:43.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:42.998+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:43.378 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:43.373+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:43.379 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:43.373+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:43.451 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:43.447+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:43.503 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:43.498+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:43.784 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:43.777+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:43.989 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:43.982+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:43.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:43.986+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:44.342 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:44.338+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:44.359 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:44.353+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:44.466 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:44.457+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:44.494 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:44.489+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:44.798 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:44.784+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:44.969 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:44.962+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:44.971 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:44.965+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:45.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:45.353+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:45.373 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:45.360+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:45.459 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:45.453+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:45.533 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:45.527+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:45.800 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:45.794+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:45.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:45.928+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:45.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:45.929+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:46.342 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:46.335+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:46.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:46.412+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:46.429 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:46.424+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:46.561 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:46.557+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:46.773 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:46.758+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:46.986 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:46.980+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:46.986 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:46.981+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:47.369 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:47.360+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:47.444 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:55:47.438+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.559401+0000 front 2019-12-11T22:54:31.570578+0000 (oldest deadline 2019-12-11T22:55:16.828505+0000) 2019-12-11T22:55:47.449 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:55:47.444+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:28.065993+0000 front 2019-12-11T22:54:28.352744+0000 (oldest deadline 2019-12-11T22:55:13.250562+0000) 2019-12-11T22:55:47.579 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:55:47.573+0000 1694f700 -1 osd.3 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.686385+0000 front 2019-12-11T22:54:29.483579+0000 (oldest deadline 2019-12-11T22:55:13.374226+0000) 2019-12-11T22:55:47.772 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:55:47.765+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.940000+0000 front 2019-12-11T22:54:31.942874+0000 (oldest deadline 2019-12-11T22:55:16.023139+0000) 2019-12-11T22:55:48.038 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:48.033+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:31.391785+0000 front 2019-12-11T22:54:31.389957+0000 (oldest deadline 2019-12-11T22:55:15.961619+0000) 2019-12-11T22:55:48.040 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:48.036+0000 1694f700 -1 osd.7 371 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:48.293 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:55:48.288+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.199:6824/10149,v1:172.21.15.199:6825/10149] for 44sec e371 v371) 2019-12-11T22:55:48.407 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:55:48.398+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.199:6828 osd.5 since back 2019-12-11T22:54:29.799863+0000 front 2019-12-11T22:54:30.413689+0000 (oldest deadline 2019-12-11T22:55:13.735443+0000) 2019-12-11T22:55:49.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:49.098+0000 1694f700 -1 osd.7 372 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:50.182 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:50.134+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:51.209 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:51.202+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:52.217 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:52.210+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:53.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:53.170+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:54.137 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:54.132+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:55.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:55.176+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:56.204 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:56.196+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:57.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:57.207+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:58.209 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:58.203+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:55:59.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:55:59.212+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:00.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:00.186+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:01.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:01.209+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:02.263 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:02.258+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:03.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:03.303+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:03.541 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:23:50.472 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.595 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:23:50.458 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.595 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:23:50.464 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.596 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:23:50.478 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.596 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:23:50.484 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.596 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:23:50.492 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.786 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:23:50.716 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.866 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:23:50.797 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:03.913 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:23:50.814 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:56:04.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:04.262+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:05.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:05.294+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:06.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:06.278+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:07.312 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:07.307+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:08.308 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:08.302+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:09.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:09.255+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:10.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:10.257+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:11.253 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:11.248+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:12.301 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:12.297+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:13.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:13.309+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:14.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:14.345+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:15.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:15.306+0000 1694f700 -1 osd.7 373 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:16.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:16.311+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:17.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:17.296+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:18.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:18.259+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:19.229 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:19.224+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:20.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:20.273+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:21.196 INFO:tasks.ceph.osd.5.smithi199.stderr:*** Caught signal (Segmentation fault) ** 2019-12-11T22:56:21.197 INFO:tasks.ceph.osd.5.smithi199.stderr: in thread 114ae700 thread_name:msgr-worker-0 2019-12-11T22:56:21.240 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:21.230+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:21.325 INFO:tasks.ceph.osd.5.smithi199.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-11T22:56:21.325 INFO:tasks.ceph.osd.5.smithi199.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-11T22:56:21.325 INFO:tasks.ceph.osd.5.smithi199.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-11T22:56:21.326 INFO:tasks.ceph.osd.5.smithi199.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-11T22:56:21.326 INFO:tasks.ceph.osd.5.smithi199.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-11T22:56:21.326 INFO:tasks.ceph.osd.5.smithi199.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-11T22:56:21.326 INFO:tasks.ceph.osd.5.smithi199.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-11T22:56:21.326 INFO:tasks.ceph.osd.5.smithi199.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-11T22:56:21.327 INFO:tasks.ceph.osd.5.smithi199.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-11T22:56:21.327 INFO:tasks.ceph.osd.5.smithi199.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-11T22:56:21.327 INFO:tasks.ceph.osd.5.smithi199.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-11T22:56:21.327 INFO:tasks.ceph.osd.5.smithi199.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-11T22:56:21.328 INFO:tasks.ceph.osd.5.smithi199.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-11T22:56:21.328 INFO:tasks.ceph.osd.5.smithi199.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-11T22:56:21.724 INFO:tasks.ceph.osd.5.smithi199.stderr:2019-12-11T22:56:21.713+0000 114ae700 -1 *** Caught signal (Segmentation fault) ** 2019-12-11T22:56:21.725 INFO:tasks.ceph.osd.5.smithi199.stderr: in thread 114ae700 thread_name:msgr-worker-0 2019-12-11T22:56:21.725 INFO:tasks.ceph.osd.5.smithi199.stderr: 2019-12-11T22:56:21.725 INFO:tasks.ceph.osd.5.smithi199.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-11T22:56:21.725 INFO:tasks.ceph.osd.5.smithi199.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-11T22:56:21.725 INFO:tasks.ceph.osd.5.smithi199.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-11T22:56:21.725 INFO:tasks.ceph.osd.5.smithi199.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-11T22:56:21.725 INFO:tasks.ceph.osd.5.smithi199.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-11T22:56:21.726 INFO:tasks.ceph.osd.5.smithi199.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-11T22:56:21.726 INFO:tasks.ceph.osd.5.smithi199.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-11T22:56:21.726 INFO:tasks.ceph.osd.5.smithi199.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-11T22:56:21.726 INFO:tasks.ceph.osd.5.smithi199.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-11T22:56:21.726 INFO:tasks.ceph.osd.5.smithi199.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-11T22:56:21.726 INFO:tasks.ceph.osd.5.smithi199.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-11T22:56:21.726 INFO:tasks.ceph.osd.5.smithi199.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-11T22:56:21.727 INFO:tasks.ceph.osd.5.smithi199.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-11T22:56:21.727 INFO:tasks.ceph.osd.5.smithi199.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-11T22:56:21.727 INFO:tasks.ceph.osd.5.smithi199.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-11T22:56:21.727 INFO:tasks.ceph.osd.5.smithi199.stderr: 2019-12-11T22:56:22.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:22.260+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:22.361 INFO:tasks.ceph.osd.5.smithi199.stderr: -14> 2019-12-11T22:56:21.713+0000 114ae700 -1 *** Caught signal (Segmentation fault) ** 2019-12-11T22:56:22.361 INFO:tasks.ceph.osd.5.smithi199.stderr: in thread 114ae700 thread_name:msgr-worker-0 2019-12-11T22:56:22.361 INFO:tasks.ceph.osd.5.smithi199.stderr: 2019-12-11T22:56:22.362 INFO:tasks.ceph.osd.5.smithi199.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-11T22:56:22.362 INFO:tasks.ceph.osd.5.smithi199.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-11T22:56:22.362 INFO:tasks.ceph.osd.5.smithi199.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-11T22:56:22.362 INFO:tasks.ceph.osd.5.smithi199.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-11T22:56:22.362 INFO:tasks.ceph.osd.5.smithi199.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-11T22:56:22.362 INFO:tasks.ceph.osd.5.smithi199.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-11T22:56:22.362 INFO:tasks.ceph.osd.5.smithi199.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-11T22:56:22.363 INFO:tasks.ceph.osd.5.smithi199.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-11T22:56:22.363 INFO:tasks.ceph.osd.5.smithi199.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-11T22:56:22.363 INFO:tasks.ceph.osd.5.smithi199.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-11T22:56:22.363 INFO:tasks.ceph.osd.5.smithi199.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-11T22:56:22.363 INFO:tasks.ceph.osd.5.smithi199.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-11T22:56:22.363 INFO:tasks.ceph.osd.5.smithi199.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-11T22:56:22.363 INFO:tasks.ceph.osd.5.smithi199.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-11T22:56:22.364 INFO:tasks.ceph.osd.5.smithi199.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-11T22:56:22.364 INFO:tasks.ceph.osd.5.smithi199.stderr: 2019-12-11T22:56:22.700 INFO:tasks.ceph.osd.5.smithi199.stderr: -1015> 2019-12-11T22:56:21.713+0000 114ae700 -1 *** Caught signal (Segmentation fault) ** 2019-12-11T22:56:22.700 INFO:tasks.ceph.osd.5.smithi199.stderr: in thread 114ae700 thread_name:msgr-worker-0 2019-12-11T22:56:22.701 INFO:tasks.ceph.osd.5.smithi199.stderr: 2019-12-11T22:56:22.701 INFO:tasks.ceph.osd.5.smithi199.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-11T22:56:22.701 INFO:tasks.ceph.osd.5.smithi199.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-11T22:56:22.701 INFO:tasks.ceph.osd.5.smithi199.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-11T22:56:22.701 INFO:tasks.ceph.osd.5.smithi199.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-11T22:56:22.701 INFO:tasks.ceph.osd.5.smithi199.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-11T22:56:22.702 INFO:tasks.ceph.osd.5.smithi199.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-11T22:56:22.702 INFO:tasks.ceph.osd.5.smithi199.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-11T22:56:22.702 INFO:tasks.ceph.osd.5.smithi199.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-11T22:56:22.702 INFO:tasks.ceph.osd.5.smithi199.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-11T22:56:22.702 INFO:tasks.ceph.osd.5.smithi199.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-11T22:56:22.702 INFO:tasks.ceph.osd.5.smithi199.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-11T22:56:22.703 INFO:tasks.ceph.osd.5.smithi199.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-11T22:56:22.703 INFO:tasks.ceph.osd.5.smithi199.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-11T22:56:22.703 INFO:tasks.ceph.osd.5.smithi199.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-11T22:56:22.703 INFO:tasks.ceph.osd.5.smithi199.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-11T22:56:22.703 INFO:tasks.ceph.osd.5.smithi199.stderr: 2019-12-11T22:56:23.242 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:23.237+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:24.202 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:24.196+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:25.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:25.170+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:25.890 INFO:tasks.ceph.osd.5.smithi199.stderr:daemon-helper: command crashed with signal 11 2019-12-11T22:56:26.220 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:26.214+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:27.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:27.216+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:28.177 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:28.170+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:28.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2019-12-11T22:56:29.216 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:29.210+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:30.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:30.228+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:31.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:31.213+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:32.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:32.217+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:33.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:33.175+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:34.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:34.159+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:34.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2019-12-11T22:56:35.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:35.145+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:36.191 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:36.184+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:37.235 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:37.229+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:38.282 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:38.276+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:39.239 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:39.231+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:40.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:40.284+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:40.483 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:40.478+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:41.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~13s 2019-12-11T22:56:41.309 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:41.303+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:41.527 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:41.522+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:42.346 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:42.341+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:42.527 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:42.523+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:43.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:43.385+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:43.565 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:43.561+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:44.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:44.440+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:44.591 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:44.587+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:45.488 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:45.483+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:45.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:45.571+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:46.532 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:46.527+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:46.537 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:46.531+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:47.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~19s 2019-12-11T22:56:47.493 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:47.489+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:47.494 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:47.490+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:47.561 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:47.555+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:48.521 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:48.515+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:48.522 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:48.517+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:48.544 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:48.536+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:49.072 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:49.066+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:49.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:49.503+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:49.510 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:49.505+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:49.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:49.575+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:49.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:49.593+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:50.097 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:50.090+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:50.535 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:50.530+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:50.535 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:50.532+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:50.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:50.596+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:50.605 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:50.598+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:50.644 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:50.640+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:50.780 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:50.773+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:51.100 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:51.095+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:51.290 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:51.282+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:51.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:51.570+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:51.576 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:51.572+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:51.593 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:51.585+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:51.593 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:51.588+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:51.680 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:51.675+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:51.835 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:51.828+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:52.148 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:52.143+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:52.272 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:52.265+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:52.549 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:52.543+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:52.550 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:52.545+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:52.570 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:52.562+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:52.571 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:52.565+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:52.716 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:52.711+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:52.828 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:52.821+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:53.116 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:53.111+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:53.273 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:53.267+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:53.514 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:53.508+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:53.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:53.510+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:53.565 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:53.559+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:53.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:53.563+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:53.682 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:53.678+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:53.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~25s 2019-12-11T22:56:53.821 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:53.816+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:54.155 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:54.151+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:54.309 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:54.302+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:54.504 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:54.500+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:54.506 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:54.502+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:54.569 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:54.563+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:54.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:54.566+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:54.719 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:54.715+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:54.813 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:54.804+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:55.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:55.123+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:55.289 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:55.280+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:55.486 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:55.481+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:55.487 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:55.483+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:55.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:55.597+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:55.605 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:55.599+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:55.754 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:55.749+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:55.780 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:55.773+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:56.096 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:56.091+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:56.339 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:56.332+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:56.511 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:56.506+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:56.511 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:56.507+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:56.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:56.582+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:56.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:56.586+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:56.743 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:56.738+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:56.744 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:56.738+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:57.098 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:57.094+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:57.312 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:57.305+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:57.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:57.461+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:57.466 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:57.462+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:57.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:57.590+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:57.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:57.592+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:57.736 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:57.728+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:57.777 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:57.772+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:58.085 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:58.080+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:58.276 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:58.264+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:58.516 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:58.512+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:58.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:58.513+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:58.599 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:58.592+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:58.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:58.594+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:58.758 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:58.751+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:56:58.779 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:58.775+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:59.106 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:56:59.102+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:56:59.318 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:56:59.312+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:56:59.552 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:59.548+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:56:59.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:56:59.550+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:56:59.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:59.602+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:56:59.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:56:59.604+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:56:59.734 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:56:59.729+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:56:59.750 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:56:59.743+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:00.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~32s 2019-12-11T22:57:00.079 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:00.075+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:00.307 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:00.300+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:00.556 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:00.551+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:00.560 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:00.556+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:00.578 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:00.570+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:00.579 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:00.573+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:00.738 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:00.733+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:00.742 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:00.735+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:01.117 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:01.113+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:01.357 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:01.349+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:01.543 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:01.533+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:01.543 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:01.534+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:01.602 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:01.589+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:01.602 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:01.592+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:01.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:01.751+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:01.792 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:01.784+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:02.103 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:02.099+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:02.340 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:02.332+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:02.548 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:02.543+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:02.551 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:02.547+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:02.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:02.625+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:02.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:02.628+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:02.728 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:02.722+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:02.777 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:02.770+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:03.069 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:03.064+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:03.362 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:03.354+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:03.537 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:03.531+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:03.539 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:03.534+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:03.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:03.640+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:03.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:03.643+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:03.701 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:03.696+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:03.746 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:03.741+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:04.105 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:04.100+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:04.414 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:04.408+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:04.498 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:04.494+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:04.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:04.497+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:04.670 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:04.664+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:04.699 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:04.693+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:04.700 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:04.695+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:04.756 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:04.750+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:05.096 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:05.090+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:05.431 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:05.425+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:05.458 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:05.452+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:05.459 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:05.454+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:05.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:05.668+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:05.677 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:05.672+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:05.718 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:05.712+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:05.723 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:05.717+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:06.106 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:06.101+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:06.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~38s 2019-12-11T22:57:06.468 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:06.462+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:06.469 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:06.464+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:06.481 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:06.474+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:06.682 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:06.675+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:06.682 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:06.677+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:06.686 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:06.680+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:06.767 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:06.762+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:07.076 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:07.070+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:07.431 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:07.426+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:07.432 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:07.429+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:07.497 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:07.481+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:07.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:07.631+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:07.638 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:07.633+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:07.688 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:07.682+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:07.746 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:07.742+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:08.083 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:08.078+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:08.394 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:08.389+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:08.396 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:08.390+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:08.515 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:08.510+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:08.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:08.598+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:08.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:08.600+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:08.690 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:08.683+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:08.728 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:08.723+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:09.077 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:09.072+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:09.439 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:09.433+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:09.440 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:09.436+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:09.543 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:09.538+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:09.640 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:09.632+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:09.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:09.635+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:09.658 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:09.651+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:09.743 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:09.738+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:10.036 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:10.032+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:10.489 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:10.480+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:10.491 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:10.484+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:10.600 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:10.592+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:10.644 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:10.635+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:10.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:10.639+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:10.661 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:10.654+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:10.764 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:10.758+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:11.042 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:11.035+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:11.471 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:11.468+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:11.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:11.470+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:11.576 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:11.570+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:11.616 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:11.609+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:11.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:11.625+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:11.634 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:11.628+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:11.800 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:11.793+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:12.013 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:12.007+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:12.477 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:12.472+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:12.478 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:12.474+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:12.543 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:12.536+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:12.637 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:12.626+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:12.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~44s 2019-12-11T22:57:12.684 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:12.660+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:12.684 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:12.663+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:12.805 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:57:12.800+0000 1694f700 -1 osd.0 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.181105+0000 front 2019-12-11T22:56:04.216620+0000 (oldest deadline 2019-12-11T22:56:48.862765+0000) 2019-12-11T22:57:13.012 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:57:13.008+0000 1694f700 -1 osd.2 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.291536+0000 front 2019-12-11T22:56:04.116386+0000 (oldest deadline 2019-12-11T22:56:48.139741+0000) 2019-12-11T22:57:13.447 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:13.443+0000 1694f700 -1 osd.3 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:02.047640+0000 front 2019-12-11T22:56:02.046925+0000 (oldest deadline 2019-12-11T22:56:47.305390+0000) 2019-12-11T22:57:13.449 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:13.444+0000 1694f700 -1 osd.3 374 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:13.583 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:57:13.577+0000 1694f700 -1 osd.4 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.963846+0000 front 2019-12-11T22:56:05.998247+0000 (oldest deadline 2019-12-11T22:56:51.154278+0000) 2019-12-11T22:57:13.656 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:57:13.650+0000 1694f700 -1 osd.6 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:05.230662+0000 front 2019-12-11T22:56:05.016998+0000 (oldest deadline 2019-12-11T22:56:50.311807+0000) 2019-12-11T22:57:13.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:13.672+0000 1694f700 -1 osd.7 374 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:56:04.908294+0000 front 2019-12-11T22:56:04.912797+0000 (oldest deadline 2019-12-11T22:56:49.599184+0000) 2019-12-11T22:57:13.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:57:13.674+0000 1694f700 -1 osd.7 374 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e371) 2019-12-11T22:57:14.731 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:57:14.680+0000 1694f700 -1 osd.3 375 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113087 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 5438773168 gen 1] snapc 0=[] ondisk+write+known_if_redirected e373) 2019-12-11T22:57:16.069 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T22:57:16.037+0000 20e25700 -1 RGWWatcher::handle_error cookie 5438773168 err (107) Transport endpoint is not connected 2019-12-11T22:57:19.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~51s 2019-12-11T22:57:25.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~57s 2019-12-11T22:57:31.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~63s 2019-12-11T22:57:37.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~69s 2019-12-11T22:57:44.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~76s 2019-12-11T22:57:50.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~82s 2019-12-11T22:57:56.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~88s 2019-12-11T22:57:58.962 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:57:58.933+0000 31786700 -1 osd.1 379 build_incremental_map_msg missing incremental map 379 2019-12-11T22:57:58.972 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:57:58.939+0000 31786700 -1 osd.1 379 build_incremental_map_msg missing incremental map 379 2019-12-11T22:58:00.279 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T22:58:00.215+0000 34f8d700 -1 osd.1 379 build_incremental_map_msg missing incremental map 379 2019-12-11T22:58:03.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~95s 2019-12-11T22:58:08.934 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:25:55.866 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:08.943 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:25:55.874 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:08.988 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:25:55.917 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:08.993 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:25:55.895 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:08.994 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:25:55.925 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:08.997 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:25:55.899 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:09.076 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:25:56.007 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:09.171 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:25:56.102 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:09.191 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:25:56.092 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:58:09.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~101s 2019-12-11T22:58:15.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~107s 2019-12-11T22:58:22.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~114s 2019-12-11T22:58:28.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:28.156+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:28.431 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:28.427+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:28.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~120s 2019-12-11T22:58:29.180 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:29.175+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:29.391 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:29.388+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:30.205 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:30.200+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:30.428 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:30.424+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:31.174 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:31.168+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:31.401 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:31.397+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:32.189 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:32.183+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:32.437 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:32.433+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:33.207 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:33.201+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:33.460 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:33.456+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:34.225 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:34.220+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:34.442 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:34.438+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:34.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~126s 2019-12-11T22:58:35.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:35.177+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:35.456 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:35.453+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:36.172 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:36.165+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:36.418 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:36.414+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:37.140 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:37.131+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:37.467 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:37.463+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:38.188 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:38.181+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:38.511 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:38.507+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:39.191 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:39.185+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:39.560 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:39.556+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:40.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:40.189+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:40.540 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:40.536+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:41.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~133s 2019-12-11T22:58:41.212 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:41.205+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:41.495 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:41.491+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:42.207 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:42.201+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:42.538 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:42.534+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:43.180 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:43.174+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:43.534 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:43.530+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:44.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:44.217+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:44.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:44.503+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:45.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:45.222+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:45.503 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:45.500+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:46.217 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:46.210+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:46.466 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:46.461+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:47.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:47.184+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:47.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~139s 2019-12-11T22:58:47.461 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:47.453+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:48.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:48.215+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:48.421 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:48.416+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:49.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:49.229+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:49.420 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:49.415+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:50.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:50.213+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:50.393 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:50.388+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:51.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:51.254+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:51.350 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:51.346+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:52.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:52.284+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:52.333 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:52.328+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:53.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:53.320+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:53.381 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:53.376+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:53.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~145s 2019-12-11T22:58:54.337 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:54.332+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:54.351 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:54.347+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:55.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:55.319+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:55.375 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:55.371+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:56.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:56.307+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:58:56.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:56.310+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:56.336 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:56.331+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:57.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:57.282+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:58:57.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:57.284+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:57.326 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:57.320+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:58:57.326 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:57.322+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:57.888 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:58:57.880+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:58:57.995 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:58:57.990+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:58:58.084 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:58:58.077+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:58:58.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:58.243+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:58:58.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:58.246+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:58.304 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:58.298+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:58:58.305 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:58.300+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:58.913 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:58:58.905+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:58:59.003 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:58:58.998+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:58:59.112 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:58:59.105+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:58:59.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:59.282+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:58:59.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:58:59.285+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:58:59.302 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:59.296+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:58:59.302 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:58:59.298+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:58:59.724 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:58:59.719+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:58:59.889 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:58:59.882+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:58:59.997 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:58:59.988+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:00.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~152s 2019-12-11T22:59:00.109 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:00.101+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:00.292 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:00.284+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:00.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:00.289+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:00.349 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:00.343+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:00.349 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:00.344+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:00.706 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:00.701+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:00.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:00.926+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:00.998 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:00.993+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:01.134 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:01.127+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:01.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:01.339+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:01.347 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:01.342+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:01.388 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:01.381+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:01.390 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:01.384+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:01.754 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:01.746+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:01.940 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:01.935+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:02.034 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:02.029+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:02.136 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:02.131+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:02.338 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:02.323+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:02.338 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:02.324+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:02.424 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:02.418+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:02.425 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:02.420+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:02.712 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:02.706+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:02.913 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:02.907+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:03.013 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:03.007+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:03.149 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:03.140+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:03.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:03.288+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:03.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:03.289+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:03.399 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:03.395+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:03.401 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:03.397+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:03.675 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:03.671+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:03.965 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:03.959+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:04.040 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:04.034+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:04.179 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:04.173+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:04.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:04.245+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:04.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:04.251+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:04.390 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:04.385+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:04.392 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:04.388+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:04.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:04.704+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:04.939 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:04.933+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:05.061 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:05.056+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:05.209 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:05.203+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:05.227 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:05.220+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:05.227 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:05.222+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:05.430 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:05.423+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:05.431 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:05.426+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:05.701 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:05.696+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:05.943 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:05.937+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:06.108 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:06.105+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:06.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:06.176+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:06.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:06.179+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:06.219 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:06.214+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:06.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~158s 2019-12-11T22:59:06.476 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:06.469+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:06.476 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:06.472+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:06.685 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:06.680+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:06.983 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:06.975+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:07.106 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:07.100+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:07.145 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:07.138+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:07.146 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:07.140+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:07.269 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:07.263+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:07.519 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:07.512+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:07.519 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:07.515+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:07.727 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:07.722+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:08.011 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:08.006+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:08.115 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:08.110+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:08.135 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:08.129+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:08.138 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:08.133+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:08.330 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:08.319+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:08.549 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:08.542+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:08.551 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:08.545+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:08.697 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:08.693+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:08.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:08.994+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:09.082 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:09.077+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:09.120 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:09.114+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:09.121 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:09.116+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:09.296 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:09.291+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:09.529 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:09.523+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:09.530 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:09.526+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:09.682 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:09.676+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:09.964 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:09.957+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:10.087 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:10.080+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:10.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:10.156+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:10.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:10.158+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:10.334 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:10.326+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:10.549 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:10.544+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:10.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:10.549+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:10.692 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:10.687+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:10.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:10.929+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:11.043 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:11.038+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:11.167 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:11.160+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:11.168 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:11.162+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:11.348 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:11.340+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:11.556 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:11.551+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:11.558 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:11.552+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:11.723 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:11.719+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:11.965 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:11.951+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:12.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:12.010+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:12.134 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:12.126+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:12.134 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:12.128+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:12.309 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:12.303+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:12.605 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:12.598+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:12.605 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:12.601+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:12.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~164s 2019-12-11T22:59:12.679 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:12.673+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:12.938 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:12.931+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:13.021 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:13.015+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:13.094 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:13.086+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:13.095 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:13.089+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:13.369 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:13.353+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:13.573 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:13.566+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:13.577 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:13.571+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:13.664 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:13.659+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:13.947 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:13.940+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:14.064 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:14.059+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:14.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:14.140+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:14.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:14.143+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:14.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:14.375+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:14.596 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:14.591+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:14.597 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:14.593+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:14.701 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:14.697+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:14.929 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:14.922+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:15.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:15.087+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:15.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:15.146+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:15.154 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:15.149+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:15.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:15.374+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:15.594 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:15.589+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:15.595 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:15.591+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:15.727 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:15.721+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:15.958 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:15.951+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:16.079 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:16.075+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:16.160 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:16.152+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:16.160 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:16.154+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:16.422 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:16.415+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:16.587 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:16.581+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:16.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:16.583+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:16.711 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:16.705+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:16.930 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:16.924+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:17.084 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:17.079+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:17.212 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:17.205+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:17.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:17.207+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:17.466 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:17.460+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:17.566 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:17.558+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:17.567 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:17.561+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:17.707 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:17.700+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:17.950 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:17.942+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:18.069 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:18.065+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:18.237 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:18.230+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:18.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:18.232+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:18.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:18.419+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:18.615 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:18.610+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:18.618 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:18.615+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:18.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:18.730+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:18.934 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:18.928+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:18.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~171s 2019-12-11T22:59:19.031 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:19.027+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:19.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:19.246+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:19.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:19.249+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:19.419 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:19.412+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:19.625 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:19.619+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:19.626 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:19.622+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:19.766 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:19.761+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:19.963 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:19.956+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:19.999 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:19.995+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:20.276 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:20.269+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:20.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:20.274+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:20.396 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:20.389+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:20.610 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:20.604+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:20.611 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:20.607+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:20.749 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:20.745+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:20.947 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:20.939+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:21.032 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:21.028+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:21.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:21.242+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:21.253 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:21.248+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:21.355 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:21.348+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:21.649 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:21.645+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:21.652 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:21.649+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:21.717 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:21.713+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:21.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:21.899+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:22.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:22.015+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:22.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:22.207+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:22.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:22.209+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:22.406 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:22.399+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:22.698 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:22.694+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:22.699 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:22.695+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:22.743 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:22.739+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:22.870 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:22.857+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:22.998 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:22.994+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:23.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:23.215+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:23.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:23.217+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:23.366 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:23.358+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:23.693 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:23.688+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:23.695 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:23.692+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:23.755 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:23.750+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:23.858 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:23.851+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:23.962 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:23.958+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:24.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:24.200+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:24.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:24.203+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:24.365 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:24.358+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:24.672 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:24.667+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:24.672 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:24.669+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:24.754 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:24.749+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:24.845 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:24.839+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:24.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:24.981+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:25.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:25.171+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:25.179 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:25.173+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:25.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~177s 2019-12-11T22:59:25.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:25.353+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:25.656 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:25.650+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:25.657 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:25.652+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:25.713 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:25.707+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:25.862 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:25.855+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:25.990 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:25.986+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:26.230 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:26.223+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:26.233 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:26.228+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:26.315 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:26.308+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:26.641 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:26.638+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:26.643 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:26.640+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:26.721 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:26.717+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:26.834 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:26.827+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:26.968 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:26.964+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:27.247 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:27.240+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:27.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:27.242+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:27.332 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:27.325+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:27.615 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:27.609+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:27.616 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:27.612+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:27.726 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:27.722+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:27.792 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:27.785+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:28.007 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:28.002+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:28.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:28.243+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:28.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:28.247+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:28.367 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T22:59:28.360+0000 1694f700 -1 osd.6 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:12.086058+0000 front 2019-12-11T22:58:12.018300+0000 (oldest deadline 2019-12-11T22:58:57.890399+0000) 2019-12-11T22:59:28.591 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:28.585+0000 1694f700 -1 osd.3 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.653445+0000 front 2019-12-11T22:58:13.646573+0000 (oldest deadline 2019-12-11T22:58:56.818114+0000) 2019-12-11T22:59:28.592 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:28.587+0000 1694f700 -1 osd.3 379 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:28.759 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T22:59:28.754+0000 1694f700 -1 osd.0 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.794342+0000 front 2019-12-11T22:58:13.775480+0000 (oldest deadline 2019-12-11T22:58:59.060596+0000) 2019-12-11T22:59:28.828 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T22:59:28.820+0000 1694f700 -1 osd.4 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:11.865395+0000 front 2019-12-11T22:58:11.858952+0000 (oldest deadline 2019-12-11T22:58:57.737641+0000) 2019-12-11T22:59:29.061 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T22:59:29.056+0000 1694f700 -1 osd.2 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.646983+0000 front 2019-12-11T22:58:13.686985+0000 (oldest deadline 2019-12-11T22:58:57.652718+0000) 2019-12-11T22:59:29.201 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T22:59:29.196+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] for 46sec e379 v379) 2019-12-11T22:59:29.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:29.248+0000 1694f700 -1 osd.7 379 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T22:58:13.585468+0000 front 2019-12-11T22:58:13.480156+0000 (oldest deadline 2019-12-11T22:58:56.092684+0000) 2019-12-11T22:59:29.257 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T22:59:29.251+0000 1694f700 -1 osd.7 379 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2.3b127d48 (undecoded) ondisk+retry+read+known_if_redirected e377) 2019-12-11T22:59:29.713 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:29.585+0000 1694f700 -1 osd.3 380 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:30.768 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T22:59:30.553+0000 1694f700 -1 osd.3 381 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:113261 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected e378) 2019-12-11T22:59:31.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~183s 2019-12-11T22:59:37.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~189s 2019-12-11T22:59:44.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~196s 2019-12-11T22:59:50.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~202s 2019-12-11T22:59:55.441 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:27:42.369 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.445 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:27:42.376 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.454 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:27:42.385 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.459 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:27:42.390 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.477 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:27:42.379 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.499 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:27:42.399 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.588 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:27:42.519 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.597 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:27:42.528 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:55.705 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:27:42.606 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T22:59:56.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~208s 2019-12-11T23:00:00.130 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:00:00.127+0000 1694f700 -1 osd.0 381 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113407 5.5 5:a93a5511:::notify.2:head [watch reconnect cookie 821300688 gen 122] snapc 0=[] ondisk+write+known_if_redirected e380) 2019-12-11T23:00:03.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~215s 2019-12-11T23:00:09.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~221s 2019-12-11T23:00:10.040 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:00:10.018+0000 30f85700 -1 osd.1 384 build_incremental_map_msg missing incremental map 384 2019-12-11T23:00:10.295 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:00:10.177+0000 35f8f700 -1 osd.1 384 build_incremental_map_msg missing incremental map 384 2019-12-11T23:00:10.304 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:00:10.202+0000 31786700 -1 osd.1 384 build_incremental_map_msg missing incremental map 384 2019-12-11T23:00:10.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:00:10.213+0000 31f87700 -1 osd.1 384 build_incremental_map_msg missing incremental map 384 2019-12-11T23:00:15.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~227s 2019-12-11T23:00:22.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~234s 2019-12-11T23:00:28.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~240s 2019-12-11T23:00:34.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~246s 2019-12-11T23:00:38.891 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:38.887+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:39.138 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:39.131+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:39.872 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:39.868+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:40.175 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:40.169+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:40.856 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:40.852+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:41.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~253s 2019-12-11T23:00:41.180 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:41.175+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:41.849 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:41.846+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:42.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:42.192+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:42.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:42.844+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:43.206 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:43.199+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:43.861 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:43.858+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:44.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:44.252+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:44.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:44.838+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:45.217 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:45.212+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:45.880 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:45.876+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:46.186 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:46.180+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:46.833 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:46.830+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:47.235 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:47.230+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:47.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~259s 2019-12-11T23:00:47.796 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:47.793+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:48.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:48.265+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:48.837 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:48.834+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:49.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:49.305+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:49.864 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:49.861+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:50.304 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:50.299+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:50.834 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:50.830+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:51.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:51.297+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:51.812 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:51.808+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:52.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:52.338+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:52.801 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:52.797+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:53.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:53.358+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:53.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~265s 2019-12-11T23:00:53.829 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:53.826+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:54.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:54.343+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:54.833 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:54.828+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:55.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:55.344+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:55.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:55.880+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:56.347 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:56.342+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:56.926 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:56.921+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:57.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:57.336+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:57.895 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:57.891+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:58.307 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:58.301+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:58.922 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:58.918+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:00:59.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:00:59.261+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:00:59.959 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:00:59.954+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:00.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~272s 2019-12-11T23:01:00.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:00.245+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:00.951 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:00.946+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:01.240 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:01.235+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:01.976 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:01.971+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:02.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:02.276+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:03.030 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:03.024+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:03.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:03.258+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:04.032 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:04.027+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:04.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:04.305+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:05.018 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:05.015+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:05.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:05.273+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:06.057 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:06.053+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:06.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:06.279+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:06.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~278s 2019-12-11T23:01:07.056 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:07.052+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:07.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:07.275+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:08.034 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:08.029+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:08.240 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:08.234+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:09.017 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:09.010+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:09.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:09.253+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:10.056 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:10.051+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:10.230 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:10.225+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:11.052 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:11.049+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:11.207 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:11.201+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:12.071 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:12.067+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:12.254 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:12.248+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:12.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~284s 2019-12-11T23:01:13.084 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:13.078+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:13.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:13.278+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:14.079 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:14.075+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:14.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:14.296+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:15.083 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:15.080+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:15.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:15.288+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:16.084 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:16.080+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:16.322 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:16.316+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:17.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:17.113+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:17.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:17.290+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:18.120 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:18.115+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:18.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:18.273+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:18.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~290s 2019-12-11T23:01:19.119 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:19.114+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:19.241 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:19.235+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:20.096 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:20.092+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:20.239 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:20.233+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:21.117 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:21.114+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:21.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:21.196+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:22.083 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:22.079+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:22.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:22.195+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:23.074 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:23.070+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:23.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:23.187+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:24.097 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:24.093+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:24.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:24.173+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:25.053 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:25.049+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:25.215 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:25.209+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:25.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~297s 2019-12-11T23:01:26.061 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:26.056+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:26.168 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:26.163+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:27.067 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:27.062+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:27.156 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:27.150+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:28.084 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:28.077+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:28.193 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:28.188+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:29.048 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:29.044+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:29.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:29.159+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:30.034 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:30.029+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:30.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:30.133+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:31.065 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:31.061+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:31.099 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:31.093+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:31.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~303s 2019-12-11T23:01:31.554 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2019-12-11T23:01:31.554 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-10-1759/qa/tasks/daemonwatchdog.py", line 38, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-10-1759/qa/tasks/daemonwatchdog.py", line 114, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-10-1759/qa/tasks/daemonwatchdog.py", line 54, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 253, in __getattr__ raise AttributeError(name) AttributeError: mounts 2019-12-11T23:01:32.073 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:32.069+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:32.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:32.136+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:33.085 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:33.080+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:33.168 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:33.163+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:34.126 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:34.121+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:34.179 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:34.174+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:35.094 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:35.090+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:35.229 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:35.223+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:36.061 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:36.057+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:36.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:36.242+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:37.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:37.095+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:37.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:37.260+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:38.066 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:38.062+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:38.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:38.280+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:39.060 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:39.056+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:39.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:39.315+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:40.091 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:40.087+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:40.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:40.334+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:41.095 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:41.090+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:41.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:41.339+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:42.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:42.097+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:42.343 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:42.336+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:43.119 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:43.113+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:43.375 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:43.369+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:44.160 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:44.154+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:44.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:44.330+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:45.145 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:45.141+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:45.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:45.346+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:46.171 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:46.166+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:46.343 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:46.337+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:47.186 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:47.181+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:47.312 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:47.306+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:48.152 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:48.142+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:48.334 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:48.329+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:49.159 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:49.153+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:49.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:49.306+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:50.128 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:50.124+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:50.347 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:50.341+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:51.105 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:51.100+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:51.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:51.344+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:52.109 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:52.103+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:52.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:52.336+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:52.617 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:52.610+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:53.069 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:53.064+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:53.349 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:53.342+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:01:53.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:53.344+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:53.624 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:53.616+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:54.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:54.097+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:01:54.105 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:54.100+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:54.374 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:54.366+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:01:54.376 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:54.369+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:54.653 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:54.640+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:55.143 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:55.137+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:01:55.146 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:55.140+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:55.203 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:01:55.198+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:01:55.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:55.379+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:01:55.387 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:55.381+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:55.605 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:01:55.596+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:01:55.699 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:55.691+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:55.866 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:01:55.852+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:01:56.157 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:56.152+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:01:56.158 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:56.154+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:56.182 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:01:56.178+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:01:56.371 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:56.364+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:01:56.378 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:56.369+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:56.648 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:01:56.642+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:01:56.684 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:56.677+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:56.828 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:01:56.820+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:01:57.129 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:57.124+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:01:57.131 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:57.126+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:57.221 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:01:57.217+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:01:57.368 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:57.361+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:01:57.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:57.363+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:57.640 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:01:57.636+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:01:57.684 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:57.677+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:57.852 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:01:57.845+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:01:58.138 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:58.131+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:01:58.139 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:58.135+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:58.267 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:01:58.264+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:01:58.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:58.363+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:01:58.370 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:58.364+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:58.691 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:01:58.685+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:01:58.719 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:58.713+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:58.846 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:01:58.839+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:01:59.095 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:59.088+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:01:59.096 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:01:59.091+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:01:59.274 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:01:59.269+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:01:59.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:59.383+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:01:59.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:01:59.385+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:01:59.691 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:01:59.682+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:01:59.773 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:01:59.768+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:01:59.828 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:01:59.822+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:00.110 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:00.103+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:00.112 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:00.107+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:00.284 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:00.274+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:00.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:00.354+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:00.361 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:00.356+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:00.663 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:00.658+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:00.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:00.732+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:00.834 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:00.829+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:01.149 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:01.144+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:01.150 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:01.146+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:01.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:01.290+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:01.407 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:01.401+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:01.410 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:01.406+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:01.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:01.705+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:01.713 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:01.707+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:01.821 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:01.815+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:02.129 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:02.124+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:02.131 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:02.126+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:02.341 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:02.333+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:02.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:02.413+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:02.422 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:02.418+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:02.684 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:02.680+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:02.695 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:02.690+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:02.836 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:02.829+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:03.138 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:03.132+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:03.140 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:03.135+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:03.332 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:03.327+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:03.420 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:03.413+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:03.421 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:03.416+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:03.700 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:03.694+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:03.746 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:03.740+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:03.801 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:03.795+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:04.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:04.093+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:04.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:04.096+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:04.343 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:04.338+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:04.398 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:04.392+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:04.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:04.394+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:04.742 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:04.737+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:04.757 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:04.747+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:04.846 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:04.840+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:05.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:05.109+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:05.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:05.112+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:05.328 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:05.322+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:05.422 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:05.416+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:05.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:05.422+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:05.770 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:05.763+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:05.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:05.786+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:05.870 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:05.864+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:06.082 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:06.074+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:06.083 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:06.078+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:06.373 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:06.368+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:06.438 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:06.431+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:06.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:06.435+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:06.813 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:06.806+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:06.816 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:06.810+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:06.897 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:06.889+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:07.086 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:07.076+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:07.086 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:07.080+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:07.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:07.390+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:07.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:07.432+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:07.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:07.434+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:07.780 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:07.773+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:07.834 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:07.830+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:07.880 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:07.870+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:08.050 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:08.042+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:08.052 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:08.046+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:08.367 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:08.362+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:08.485 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:08.478+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:08.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:08.480+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:08.748 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:08.741+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:08.820+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:08.882 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:08.875+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:09.075 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:09.068+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:09.076 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:09.072+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:09.405 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:09.399+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:09.532 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:09.525+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:09.535 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:09.530+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:09.763 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:09.754+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:09.840 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:09.836+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:09.890 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:09.884+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:10.071 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:10.065+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:10.073 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:10.068+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:10.418 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:10.414+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:10.541 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:10.533+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:10.542 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:10.535+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:10.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:10.718+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:10.892 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:10.888+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:10.908 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:10.900+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:11.106 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:11.099+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:11.107 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:11.102+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:11.374 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:11.368+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:11.514 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:11.506+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:11.515 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:11.509+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:11.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:11.729+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:11.901 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:11.895+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:11.939 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:11.930+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:12.148 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:12.143+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:12.149 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:12.145+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:12.394 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:12.390+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:12.498 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:12.491+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:12.499 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:12.493+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:12.724 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:12.714+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:12.855 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:12.851+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:12.908 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:12.902+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:13.125 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:13.111+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:13.127 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:13.114+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:13.412 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:13.407+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:13.535 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:13.529+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:13.537 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:13.533+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:13.764 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:13.757+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:13.871 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:13.867+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:13.910 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:13.904+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:14.091 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:14.084+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:14.092 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:14.087+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:14.384 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:14.380+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:14.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:14.493+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:14.504 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:14.496+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:14.793 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:14.785+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:14.908 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:14.896+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:14.928 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:14.922+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:15.115 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:15.109+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:15.116 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:15.112+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:15.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:15.402+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:15.456 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:15.450+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:15.460 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:15.454+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:15.829 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:15.822+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:15.884 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:15.879+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:15.891 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:15.884+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:16.115 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:16.108+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:16.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:16.112+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:16.377 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:16.374+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:16.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:16.503+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:16.512 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:16.507+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:16.815 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:16.807+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:16.879 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:16.873+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:16.929 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:16.917+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:17.154 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:17.149+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:17.157 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:17.151+0000 1694f700 -1 osd.3 385 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4518.0:113474 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 385976240 gen 3] snapc 0=[] ondisk+write+known_if_redirected e383) 2019-12-11T23:02:17.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:02:17.414+0000 1694f700 -1 osd.2 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.896325+0000 front 2019-12-11T23:01:10.893947+0000 (oldest deadline 2019-12-11T23:01:54.924967+0000) 2019-12-11T23:02:17.503 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:17.495+0000 1694f700 -1 osd.7 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.349951+0000 front 2019-12-11T23:01:10.366874+0000 (oldest deadline 2019-12-11T23:01:53.216321+0000) 2019-12-11T23:02:17.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:02:17.500+0000 1694f700 -1 osd.7 385 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:112983 2.8s0 2:12be48dc:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.671_bcc3577a270e48528c51295a818ce2b2:head [getxattrs out=1000b,stat out=16b,read 0~141] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e383) 2019-12-11T23:02:17.797 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:02:17.790+0000 1694f700 -1 osd.6 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.517143+0000 front 2019-12-11T23:01:09.045703+0000 (oldest deadline 2019-12-11T23:01:52.519001+0000) 2019-12-11T23:02:17.864 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:17.860+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:17.902 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:17.894+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:18.164 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:18.158+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:18.625 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:18.617+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:18.856 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:02:18.849+0000 1694f700 -1 osd.0 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.167649+0000 front 2019-12-11T23:01:10.122564+0000 (oldest deadline 2019-12-11T23:01:54.796989+0000) 2019-12-11T23:02:18.918 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:18.907+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:01:09.687102+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:19.041 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-11T23:02:18.950+0000 20e25700 -1 RGWWatcher::handle_error cookie 385976240 err (107) Transport endpoint is not connected 2019-12-11T23:02:19.206 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:19.202+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:19.614 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:19.611+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:19.950 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:02:19.934+0000 1694f700 -1 osd.4 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:09.679323+0000 front 2019-12-11T23:02:19.925783+0000 (oldest deadline 2019-12-11T23:01:55.556889+0000) 2019-12-11T23:02:20.192 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:20.186+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:20.629 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:20.625+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:21.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:02:21.230+0000 1694f700 -1 osd.3 385 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2019-12-11T23:01:10.428690+0000 front 2019-12-11T23:01:10.442064+0000 (oldest deadline 2019-12-11T23:01:53.263410+0000) 2019-12-11T23:02:21.585 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:21.581+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:22.613 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:22.606+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:23.662 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:23.658+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:24.686 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:24.682+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:25.664 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:25.660+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:26.697 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:26.691+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:27.722 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:27.717+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:28.732 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:28.724+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:29.687 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:29.682+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:30.666 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:30.661+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:31.698 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:31.693+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:32.724 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:32.718+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:33.020 INFO:teuthology.orchestra.run.smithi079.stderr:testIfMatch (test.functional.tests.TestFileComparison) ... ok 2019-12-11T23:02:33.175 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:33.171+0000 35f8f700 -1 osd.1 385 build_incremental_map_msg missing incremental map 379 2019-12-11T23:02:33.189 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:33.181+0000 35f8f700 -1 osd.1 385 build_incremental_map_msg missing incremental map 384 2019-12-11T23:02:33.750 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:33.746+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:34.714 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:34.711+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:35.712 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:35.708+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:36.763 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:36.760+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:37.752 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:37.748+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:38.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:38.749+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:39.782 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:39.778+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:40.790 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:40.786+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:41.806 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:41.802+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:42.762 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:42.759+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:43.757 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:43.746+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:44.756 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:44.751+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:45.790 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:45.786+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:46.745 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:46.742+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:47.729 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:47.724+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:48.702 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:48.699+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:49.669 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:49.664+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:50.707 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:50.704+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:51.688 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:51.684+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:52.684 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:52.679+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:53.721 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:53.715+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:54.681 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:54.673+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:55.652 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:55.647+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:56.614 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:56.610+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:57.610 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:57.606+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:58.648 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:58.643+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:02:59.665 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:02:59.662+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:00.700 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:00.696+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:01.684 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:01.680+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:02.690 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:02.685+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:02.772 INFO:teuthology.orchestra.run.smithi079.stderr:testIfMatchAndUnmodified (test.functional.tests.TestFileComparison) ... ok 2019-12-11T23:03:03.716 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:03.712+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:04.738 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:04.732+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:05.754 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:05.750+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:06.732 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:06.729+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:07.756 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:07.752+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:08.785 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:08.782+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:09.761 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:09.757+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:10.787 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:10.782+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:11.775 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:11.770+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:12.740 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:12.735+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:13.730 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:13.725+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:14.756 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:14.752+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:15.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:15.774+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:16.810 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:16.806+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:17.859 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:17.855+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:18.846 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:18.842+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:19.842 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:19.838+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:20.880 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:20.876+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:21.879 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:21.876+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:22.902 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:22.899+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:23.117 INFO:teuthology.orchestra.run.smithi079.stderr:testIfModifiedSince (test.functional.tests.TestFileComparison) ... ok 2019-12-11T23:03:23.939 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:23.936+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:24.991 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:24.986+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:26.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:26.012+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:27.055 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:27.051+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:28.064 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:28.060+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:29.052 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:29.048+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:30.073 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:30.069+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:31.038 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:31.033+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:32.007 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:32.004+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:33.037 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:33.033+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:34.020 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:34.016+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:35.061 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:35.057+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:36.053 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:36.050+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:37.006 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:37.001+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:38.031 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:38.028+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:39.023 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:39.018+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:40.021 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:40.017+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:40.978 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:40.975+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:41.960 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:41.956+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:43.006 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:43.003+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:43.985 INFO:teuthology.orchestra.run.smithi079.stderr:testIfNoneMatch (test.functional.tests.TestFileComparison) ... ok 2019-12-11T23:03:43.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:43.989+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:45.002 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:44.996+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:46.001 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:45.997+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:46.997 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:46.993+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:47.961 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:47.957+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:48.941 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:48.936+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:49.930 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:49.926+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:50.942 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:50.937+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:51.919 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:51.914+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:52.926 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:52.922+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:53.900 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:53.896+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:54.898 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:54.891+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:55.245 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:31:42.176 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.253 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:31:42.185 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.263 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:31:42.194 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.268 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:31:42.199 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.276 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:31:42.178 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.301 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:31:42.202 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.613 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:31:42.515 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.618 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:31:42.550 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.628 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:31:42.559 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:03:55.872 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:55.867+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:56.918 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:56.914+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:57.889 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:57.885+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:58.881 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:58.877+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:03:59.901 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:03:59.896+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:00.941 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:00.936+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:01.914 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:01.910+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:02.897 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:02.894+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:03.642 INFO:teuthology.orchestra.run.smithi079.stderr:testIfUnmodifiedSince (test.functional.tests.TestFileComparison) ... ok 2019-12-11T23:04:03.863 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:03.860+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:04.847 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:04.844+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:05.432 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:31:52.363 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.486 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:31:52.417 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.520 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:31:52.451 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.523 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:31:52.454 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.540 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:31:52.442 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.546 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:31:52.476 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.546 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:31:52.447 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.585 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:31:52.509 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.589 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:31:52.490 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:04:05.888 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:05.884+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:06.920 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:06.916+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:07.948 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:07.944+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:08.963 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:08.960+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:10.007 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:10.002+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:11.042 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:11.038+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:12.052 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:12.049+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:13.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:13.006+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:14.035 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:14.031+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:15.043 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:15.039+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:16.032 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:16.028+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:17.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:17.044+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:18.062 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:18.059+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:19.037 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:19.032+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:20.072 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:20.068+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:21.047 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:21.042+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:22.075 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:22.071+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:23.051 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:23.047+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:24.046 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:24.042+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:25.005 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:25.002+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:26.037 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:26.034+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:27.025 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:27.021+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:28.043 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:28.039+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:29.014 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:29.008+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:29.981 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:29.977+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:30.995 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:30.991+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:32.022 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:32.019+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:33.036 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:33.033+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:34.036 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:34.032+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:35.070 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:35.066+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:36.111 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:36.099+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:37.155 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:37.150+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:38.191 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:38.188+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:39.230 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:39.223+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:40.238 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:40.234+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:41.200 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:41.196+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:42.232 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:42.227+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:43.216 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:43.211+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:44.184 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:44.178+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:45.211 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:45.207+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:46.218 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:46.214+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:47.249 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:47.245+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:48.219 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:48.216+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:49.216 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:49.212+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:50.210 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:50.198+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:51.210 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:51.205+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:52.188 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:52.184+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:53.191 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:53.187+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:54.213 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:54.209+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:55.197 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:55.193+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:56.170 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:56.166+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:57.157 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:57.153+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:58.131 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:58.127+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:04:59.127 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:04:59.123+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:00.170 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:00.166+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:01.169 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:01.161+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:02.216 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:02.212+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:03.214 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:03.210+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:04.237 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:04.233+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:05.198 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:05.194+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:06.186 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:06.181+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:07.148 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:07.144+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:08.147 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:08.141+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:09.129 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:09.125+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:10.115 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:10.112+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:11.116 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:11.112+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:12.155 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:12.151+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:13.124 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:13.120+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:14.134 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:14.131+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:15.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:15.105+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:16.123 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:16.120+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:17.093 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:17.090+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:18.086 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:18.082+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:19.125 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:19.121+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:20.099 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:20.096+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:21.061 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:21.057+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:22.086 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:22.083+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:23.123 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:23.118+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:24.115 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:24.110+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:25.108 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:25.103+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:26.076 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:26.072+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:27.107 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:27.102+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:28.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:28.082+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:28.815 INFO:teuthology.orchestra.run.smithi079.stderr:testIfMatch (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-11T23:05:29.085 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:29.081+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:30.099 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:30.096+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:31.052 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:31.049+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:32.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:32.086+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:33.138 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:33.134+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:34.157 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:34.153+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:35.181 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:35.176+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:36.232 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:36.228+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:37.288 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:37.284+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:38.265 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:38.261+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:39.303 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:39.299+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:40.322 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:40.318+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:41.355 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:41.351+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:42.314 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:42.310+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:43.312 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:43.308+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:44.345 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:44.341+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:45.327 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:45.322+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:46.317 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:46.313+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:47.290 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:47.286+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:48.326 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:48.322+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:49.305 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:49.301+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:50.351 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:50.347+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:51.353 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:51.349+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:52.408 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:52.404+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:53.447 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:53.442+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:54.476 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:54.473+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:55.526 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:55.519+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:56.532 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:56.529+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:57.592 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:57.589+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:58.609 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:58.602+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:05:59.642 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:05:59.639+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:00.601 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:00.596+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:00.982 INFO:teuthology.orchestra.run.smithi079.stderr:testIfMatchAndUnmodified (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-11T23:06:01.568 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:01.564+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:02.589 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:02.585+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:03.612 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:03.608+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:04.642 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:04.638+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:05.599 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:05.594+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:06.553 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:06.548+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:07.590 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:07.585+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:08.619 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:08.614+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:09.609 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:09.603+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:10.570 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:10.565+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:11.620 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:11.615+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:12.604 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:12.599+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:13.562 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:13.558+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:14.556 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:14.552+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:15.575 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:15.572+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:16.554 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:16.549+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:17.549 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:17.546+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:18.549 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:18.541+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:19.512 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:19.509+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:20.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:20.470+0000 1694f700 -1 osd.1 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:22.031 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:21.979+0000 1694f700 -1 osd.1 386 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:23.144 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:23.134+0000 1694f700 -1 osd.1 387 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:24.224 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:24.194+0000 1694f700 -1 osd.1 388 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:25.613 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:25.464+0000 1694f700 -1 osd.1 389 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:26.636 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:26.547+0000 1694f700 -1 osd.1 390 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:27.757 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:06:27.631+0000 1694f700 -1 osd.1 390 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4498.0:88523 7.0 7:0651570b:lc::lc.14:head [call lock.lock in=50b] snapc 0=[] RETRY=67 ondisk+retry+write+known_if_redirected e384) 2019-12-11T23:06:28.428 INFO:teuthology.orchestra.run.smithi079.stderr:testIfModifiedSince (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-11T23:06:36.226 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:34:23.158 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.236 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:34:23.162 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.265 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:34:23.196 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.274 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:34:23.176 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.275 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:34:23.177 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.281 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:34:23.212 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.462 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:34:23.364 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.553 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:34:23.483 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:06:36.617 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:34:23.545 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:01.694 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:01.687+0000 1694f700 -1 osd.6 415 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:02.787 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:02.695+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:03.675 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:03.669+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:04.640 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:04.634+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:05.680 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:05.674+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:06.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:06.715+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:07.708 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:07.702+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:08.689 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:08.683+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:09.724 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:09.718+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:10.720 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:10.715+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:11.526 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:11.509+0000 1694f700 -1 osd.7 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:11.711 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:11.705+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:12.569 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:12.561+0000 1694f700 -1 osd.7 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:12.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:12.712+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:13.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:13.625+0000 1694f700 -1 osd.7 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:13.713 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:13.705+0000 1694f700 -1 osd.6 416 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:14.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:14.607+0000 1694f700 -1 osd.7 417 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:14.744 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:14.737+0000 1694f700 -1 osd.6 417 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:15.730 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:15.706+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:15.796 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:15.678+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:16.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:16.643+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:16.733 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:16.728+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:17.504 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:35:04.436 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.525 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:35:04.444 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.539 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:35:04.439 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.539 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:35:04.453 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.540 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:35:04.460 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.555 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:35:04.456 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.616 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:35:04.547 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.641 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:35:04.572 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:17.653+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:17.754 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:35:04.655 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:07:17.781 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:17.772+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:18.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:18.614+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:18.732 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:18.728+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:19.653 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:19.648+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:19.754 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:19.747+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:20.660 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:20.655+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:20.779 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:20.774+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:21.656 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:21.651+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:21.739 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:21.731+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:22.734 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:22.729+0000 1694f700 -1 osd.7 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:22.784 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:22.779+0000 1694f700 -1 osd.6 418 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:23.714 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:23.708+0000 1694f700 -1 osd.7 419 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:23.810 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:23.805+0000 1694f700 -1 osd.6 419 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:24.835 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:24.831+0000 1694f700 -1 osd.6 420 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:24.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:24.926+0000 1694f700 -1 osd.7 420 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:25.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:25.785+0000 1694f700 -1 osd.6 420 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:25.955 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:25.945+0000 1694f700 -1 osd.7 420 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:26.756 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:26.751+0000 1694f700 -1 osd.6 420 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:26.913 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:26.907+0000 1694f700 -1 osd.7 420 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:44.527034+0000 front 2019-12-11T23:06:45.106204+0000 (oldest deadline 2019-12-11T23:07:26.126720+0000) 2019-12-11T23:07:26.913 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:26.908+0000 1694f700 -1 osd.7 420 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:27.779 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:27.773+0000 1694f700 -1 osd.6 420 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:27.955 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:07:27.951+0000 1694f700 -1 osd.3 421 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:45.393320+0000 front 2019-12-11T23:06:45.433563+0000 (oldest deadline 2019-12-11T23:07:27.408658+0000) 2019-12-11T23:07:28.037 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:27.971+0000 1694f700 -1 osd.7 421 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:44.527034+0000 front 2019-12-11T23:06:45.106204+0000 (oldest deadline 2019-12-11T23:07:26.126720+0000) 2019-12-11T23:07:28.066 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:28.001+0000 1694f700 -1 osd.7 421 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:28.758 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:28.751+0000 1694f700 -1 osd.6 421 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:44.489685+0000 front 2019-12-11T23:06:44.491057+0000 (oldest deadline 2019-12-11T23:07:28.273513+0000) 2019-12-11T23:07:28.759 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:28.753+0000 1694f700 -1 osd.6 421 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:28.941 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:07:28.938+0000 1694f700 -1 osd.3 421 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:45.393320+0000 front 2019-12-11T23:06:45.433563+0000 (oldest deadline 2019-12-11T23:07:27.408658+0000) 2019-12-11T23:07:29.162 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:29.150+0000 1694f700 -1 osd.7 422 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:44.527034+0000 front 2019-12-11T23:06:45.106204+0000 (oldest deadline 2019-12-11T23:07:26.126720+0000) 2019-12-11T23:07:29.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:29.152+0000 1694f700 -1 osd.7 422 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:29.400 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:07:29.396+0000 1694f700 -1 osd.1 422 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:46.221517+0000 front 2019-12-11T23:06:46.255532+0000 (oldest deadline 2019-12-11T23:07:29.169247+0000) 2019-12-11T23:07:29.964 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:07:29.959+0000 1694f700 -1 osd.3 422 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:06:45.393320+0000 front 2019-12-11T23:06:45.433563+0000 (oldest deadline 2019-12-11T23:07:27.408658+0000) 2019-12-11T23:07:30.005 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:29.995+0000 1694f700 -1 osd.6 423 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:30.373 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:30.239+0000 1694f700 -1 osd.7 423 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:31.433 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:07:31.311+0000 1694f700 -1 osd.6 424 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88627 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e390) 2019-12-11T23:07:31.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:31.556+0000 1694f700 -1 osd.7 424 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:07:32.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:07:32.592+0000 1694f700 -1 osd.7 424 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115117 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e399) 2019-12-11T23:08:29.005 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:08:28.998+0000 1694f700 -1 osd.6 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:42.892671+0000 front 2019-12-11T23:07:43.048247+0000 (oldest deadline 2019-12-11T23:08:28.050750+0000) 2019-12-11T23:08:29.111 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:08:29.104+0000 1694f700 -1 osd.4 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:47.846048+0000 front 2019-12-11T23:07:47.863888+0000 (oldest deadline 2019-12-11T23:08:28.255236+0000) 2019-12-11T23:08:29.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:08:29.179+0000 1694f700 -1 osd.7 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:46.760901+0000 front 2019-12-11T23:07:46.871380+0000 (oldest deadline 2019-12-11T23:08:29.032037+0000) 2019-12-11T23:08:30.041 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:08:30.033+0000 1694f700 -1 osd.6 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:42.892671+0000 front 2019-12-11T23:07:43.048247+0000 (oldest deadline 2019-12-11T23:08:28.050750+0000) 2019-12-11T23:08:30.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:08:30.117+0000 1694f700 -1 osd.4 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:47.846048+0000 front 2019-12-11T23:07:47.863888+0000 (oldest deadline 2019-12-11T23:08:28.255236+0000) 2019-12-11T23:08:30.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:08:30.135+0000 1694f700 -1 osd.7 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:46.760901+0000 front 2019-12-11T23:07:46.871380+0000 (oldest deadline 2019-12-11T23:08:29.032037+0000) 2019-12-11T23:08:30.806 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:08:30.802+0000 1694f700 -1 osd.2 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:45.400369+0000 front 2019-12-11T23:07:45.401348+0000 (oldest deadline 2019-12-11T23:08:30.094042+0000) 2019-12-11T23:08:31.067 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:08:31.060+0000 1694f700 -1 osd.6 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:42.892671+0000 front 2019-12-11T23:07:43.048247+0000 (oldest deadline 2019-12-11T23:08:28.050750+0000) 2019-12-11T23:08:31.117 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:08:31.110+0000 1694f700 -1 osd.4 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:47.846048+0000 front 2019-12-11T23:07:47.863888+0000 (oldest deadline 2019-12-11T23:08:28.255236+0000) 2019-12-11T23:08:31.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:08:31.135+0000 1694f700 -1 osd.7 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:46.760901+0000 front 2019-12-11T23:07:46.871380+0000 (oldest deadline 2019-12-11T23:08:29.032037+0000) 2019-12-11T23:08:31.788 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:08:31.771+0000 1694f700 -1 osd.2 430 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:07:45.400369+0000 front 2019-12-11T23:07:45.401348+0000 (oldest deadline 2019-12-11T23:08:30.094042+0000) 2019-12-11T23:08:47.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:08:47.231+0000 1694f700 -1 osd.0 430 get_health_metrics reporting 4 slow ops, oldest is MOSDPGRecoveryDeleteReply(3.17 e390,386 3:eb4dcc45:::.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.364:head,138'72) 2019-12-11T23:09:55.356 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:09:55.338+0000 3578e700 -1 osd.3 463 build_incremental_map_msg missing incremental map 463 2019-12-11T23:09:55.601 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:09:55.590+0000 35f8f700 -1 osd.3 463 build_incremental_map_msg missing incremental map 463 2019-12-11T23:09:55.627 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:09:55.592+0000 35f8f700 -1 osd.3 463 build_incremental_map_msg missing incremental map 463 2019-12-11T23:09:55.680 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:09:55.659+0000 35f8f700 -1 osd.3 463 build_incremental_map_msg missing incremental map 463 2019-12-11T23:09:55.731 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:09:55.689+0000 31f87700 -1 osd.3 463 build_incremental_map_msg missing incremental map 463 2019-12-11T23:09:55.877 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:09:55.708+0000 38f95700 -1 osd.3 463 build_incremental_map_msg missing incremental map 463 2019-12-11T23:09:57.543 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:09:57.538+0000 3578e700 -1 osd.3 464 build_incremental_map_msg missing incremental map 463 2019-12-11T23:10:08.958 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:37:55.889 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:08.963 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:37:55.894 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:08.970 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:37:55.902 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:08.973 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:37:55.904 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:08.976 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:37:55.878 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:08.987 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:37:55.890 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:09.299 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:37:56.231 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:09.301 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:37:56.204 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:09.370 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:37:56.301 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:10:11.555 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:10:11.552+0000 32f89700 -1 osd.3 469 build_incremental_map_msg missing incremental map 463 2019-12-11T23:10:15.967 INFO:teuthology.orchestra.run.smithi079.stderr:testIfNoneMatch (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-11T23:10:46.188 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:46.183+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:47.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:47.180+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:48.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:48.217+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:49.244 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:49.240+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:50.243 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:50.239+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:51.208 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:51.203+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:52.175 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:52.170+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:53.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:53.215+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:54.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:54.197+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:55.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:55.187+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:56.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:56.208+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:56.514 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:10:56.507+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:10:57.078 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:10:57.072+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:10:57.231 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:57.224+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:10:57.232 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:57.227+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:57.557 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:10:57.550+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:10:58.039 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:10:58.030+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:10:58.218 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:58.209+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:10:58.218 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:58.211+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:58.597 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:10:58.590+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:10:59.012 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:10:59.003+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:10:59.230 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:59.223+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:10:59.231 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:10:59.226+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:10:59.560 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:10:59.554+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:10:59.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:10:59.874+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:00.041 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:00.033+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:00.186 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:00.179+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:00.187 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:00.182+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:00.390 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:00.384+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:00.459 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:00.455+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:00.560 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:00.553+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:00.871 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:00.867+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:01.072 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:01.064+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:01.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:01.141+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:01.151 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:01.143+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:01.414 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:01.409+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:01.425 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:01.419+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:01.542 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:01.533+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:01.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:01.821+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:02.045 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:02.037+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:02.142 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:02.134+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:02.143 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:02.137+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:02.424 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:02.419+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:02.442 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:02.438+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:02.513 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:02.507+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:02.799 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:02.790+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:03.016 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:03.009+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:03.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:03.187+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:03.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:03.190+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:03.396 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:03.392+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:03.436 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:03.432+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:03.552 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:03.545+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:03.759 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:03.755+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:04.034 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:04.022+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:04.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:04.175+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:04.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:04.178+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:04.365 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:04.360+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:04.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:04.452+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:04.550 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:04.544+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:04.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:04.713+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:04.998 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:04.990+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:05.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:05.185+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:05.193 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:05.188+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:05.364 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:05.360+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:05.469 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:05.465+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:05.575 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:05.569+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:05.717 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:05.711+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:06.024 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:06.017+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:06.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:06.187+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:06.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:06.190+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:06.324 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:06.319+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:06.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:06.472+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:06.610 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:06.601+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:06.675 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:06.669+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:07.016 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:07.009+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:07.171 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:07.165+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:07.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:07.166+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:07.373 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:07.369+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:07.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:07.484+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:07.605 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:07.597+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:07.674 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:07.669+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:07.975 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:07.969+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:08.153 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:08.146+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:08.154 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:08.149+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:08.359 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:08.354+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:08.535 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:08.531+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:08.601 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:08.594+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:08.708 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:08.704+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:08.938 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:08.932+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:09.174 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:09.167+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:09.175 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:09.170+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:09.357 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:09.352+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:09.522 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:09.517+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:09.639 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:09.632+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:09.703 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:09.698+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:09.965 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:09.958+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:10.147 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:10.140+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:10.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:10.143+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:10.401 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:10.396+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:10.476 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:10.470+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:10.645 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:10.639+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:10.724 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:10.719+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:10.920 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:10.914+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:11.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:11.142+0000 1694f700 -1 osd.7 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:13.914773+0000 front 2019-12-11T23:10:15.539153+0000 (oldest deadline 2019-12-11T23:10:56.768295+0000) 2019-12-11T23:11:11.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:11.144+0000 1694f700 -1 osd.7 470 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:11:11.419 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:11:11.414+0000 1694f700 -1 osd.1 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.290865+0000 front 2019-12-11T23:10:14.339393+0000 (oldest deadline 2019-12-11T23:10:59.562717+0000) 2019-12-11T23:11:11.525 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:11:11.520+0000 1694f700 -1 osd.2 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:15.494354+0000 front 2019-12-11T23:10:14.753530+0000 (oldest deadline 2019-12-11T23:10:59.681175+0000) 2019-12-11T23:11:11.677 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:11:11.670+0000 1694f700 -1 osd.4 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:11.382912+0000 front 2019-12-11T23:10:11.342631+0000 (oldest deadline 2019-12-11T23:10:55.748661+0000) 2019-12-11T23:11:11.711 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:11:11.706+0000 1694f700 -1 osd.3 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:14.768178+0000 front 2019-12-11T23:10:15.550756+0000 (oldest deadline 2019-12-11T23:10:59.656652+0000) 2019-12-11T23:11:11.959 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:11:11.953+0000 1694f700 -1 osd.6 470 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:10:12.273489+0000 front 2019-12-11T23:10:12.121473+0000 (oldest deadline 2019-12-11T23:10:56.729944+0000) 2019-12-11T23:11:12.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:11:12.264+0000 1694f700 -1 osd.7 471 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] ondisk+read+known_if_redirected e470) 2019-12-11T23:12:12.952 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:12.946+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:13.322 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:13.319+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:13.948 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:13.943+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:13.982 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:13.978+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:14.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:14.298+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:14.370 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:14.359+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:14.940 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:14.937+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:14.972 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:14.966+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:15.272 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:15.267+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:15.399 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:15.393+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:15.960 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:15.954+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:15.989 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:15.984+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:16.289 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:16.282+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:16.366 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:16.361+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:16.609 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:16.603+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:17.004 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:16.998+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:17.034 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:17.030+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:17.282 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:17.276+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:17.409 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:17.405+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:17.571 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:17.566+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:18.017 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:18.011+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:18.078 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:18.074+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:18.250 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:18.244+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:18.453 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:18.449+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:18.616 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:18.609+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:19.007 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:19.001+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:19.086 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:19.081+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:19.245 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:19.240+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:19.453 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:19.449+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:19.598 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:19.592+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:19.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:19.966+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:20.135 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:20.131+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:20.230 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:20.223+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:20.409 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:20.405+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:20.572 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:20.566+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:20.972 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:20.967+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:21.108 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:21.105+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:21.243 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:21.236+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:21.422 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:21.418+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:21.542 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:21.536+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:21.939 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:21.934+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:22.072 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:22.069+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:22.277 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:22.270+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:22.451 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:22.447+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:22.539 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:22.532+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:22.985 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:22.980+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:23.075 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:23.070+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:23.312 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:23.307+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:23.450 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:23.445+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:23.508 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:23.502+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:23.955 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:23.948+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:24.036 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:24.030+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:24.093 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:24.089+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:24.360 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:24.356+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:24.429 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:24.425+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:24.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:24.459+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:24.980 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:24.976+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:25.054 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:25.049+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:25.076 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:25.072+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:25.406 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:25.400+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:25.456 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:25.449+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:25.488 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:25.482+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:25.943 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:25.938+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:26.039 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:26.034+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:26.059 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:26.053+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:26.427 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:26.422+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:26.476 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:26.472+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:26.500 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:26.494+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:26.977 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:26.972+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:27.084 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:27.080+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:27.111 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:27.106+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:27.439 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:27.433+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:27.457 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:27.452+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:27.486 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:27.480+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:28.007 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:28.003+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:28.064 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:28.059+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:28.161 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:28.156+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:28.448 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:28.444+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:28.474 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:28.467+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:28.476 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:28.462+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:29.014 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:29.010+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:29.081 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:29.077+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:29.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:29.208+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:29.404 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:29.399+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:29.434 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:29.429+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:29.529 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:29.523+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:30.041 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:30.038+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:30.063 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:30.060+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:30.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:30.171+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:30.424 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:30.418+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:30.427 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:30.423+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:30.510 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:30.504+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:31.089 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:31.085+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:31.108 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:31.105+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:31.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:31.164+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:31.450 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:31.445+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:31.456 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:31.450+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:31.504 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:31.499+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:32.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:32.087+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:32.135 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:32.132+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:32.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:32.136+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:32.428 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:32.423+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:32.459 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:32.454+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:32.483 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:32.477+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:33.059 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:33.055+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:33.102 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:33.097+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:33.174 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:33.169+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:33.446 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:33.439+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:33.448 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:33.444+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:33.454 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:33.448+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:34.104 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:34.099+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:34.108 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:34.104+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:34.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:34.175+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:34.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:34.453+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:34.458 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:34.451+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:34.498 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:34.492+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:35.110 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:35.107+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:35.136 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:35.132+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:35.172 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:35.163+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:35.492 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:35.485+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:35.504 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:35.495+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:35.539 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:35.534+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:36.135 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:36.129+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:36.137 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:36.134+0000 1694f700 -1 osd.0 476 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:12:36.153 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:36.147+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:36.155 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:36.150+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:36.485 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:36.479+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:36.532 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:36.526+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:36.547 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:36.542+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:37.096 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:37.091+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:37.097 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:37.093+0000 1694f700 -1 osd.0 476 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:12:37.161 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:37.157+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:37.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:37.177+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:37.498 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:37.492+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:37.531 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:37.527+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:37.540 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:37.534+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:38.100 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:38.094+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:38.109 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:38.106+0000 1694f700 -1 osd.0 476 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:12:38.144 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:38.138+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:38.145 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:38.140+0000 1694f700 -1 osd.7 476 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e475) 2019-12-11T23:12:38.188 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:38.185+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:38.477 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:38.471+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:38.552 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:38.548+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:38.569 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:38.563+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:39.110 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:39.103+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:39.110 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:39.105+0000 1694f700 -1 osd.7 476 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e475) 2019-12-11T23:12:39.142 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:39.137+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:39.145 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:39.141+0000 1694f700 -1 osd.0 476 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:12:39.220 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:39.215+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:39.449 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:39.444+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:39.554 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:39.548+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:39.555 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:39.551+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:40.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:40.065+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:40.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:40.067+0000 1694f700 -1 osd.7 476 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e475) 2019-12-11T23:12:40.178 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:40.174+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:40.179 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:40.175+0000 1694f700 -1 osd.0 476 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:12:40.226 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:40.220+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:40.462 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:40.457+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:40.566 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:40.561+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:40.596 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:40.592+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:41.077 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:41.071+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:41.078 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:41.073+0000 1694f700 -1 osd.7 476 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e475) 2019-12-11T23:12:41.194 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:41.191+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:41.217 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:41.212+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:41.217 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:41.214+0000 1694f700 -1 osd.0 476 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:12:41.484 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:12:41.478+0000 1694f700 -1 osd.4 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:28.451941+0000 front 2019-12-11T23:11:28.450728+0000 (oldest deadline 2019-12-11T23:12:13.726735+0000) 2019-12-11T23:12:41.622 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:12:41.616+0000 1694f700 -1 osd.6 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:30.036889+0000 front 2019-12-11T23:11:30.006386+0000 (oldest deadline 2019-12-11T23:12:15.877633+0000) 2019-12-11T23:12:41.644 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:12:41.640+0000 1694f700 -1 osd.1 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.149122+0000 front 2019-12-11T23:11:27.109030+0000 (oldest deadline 2019-12-11T23:12:12.970364+0000) 2019-12-11T23:12:42.122 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:42.116+0000 1694f700 -1 osd.7 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:27.008810+0000 front 2019-12-11T23:11:26.862067+0000 (oldest deadline 2019-12-11T23:12:12.141487+0000) 2019-12-11T23:12:42.123 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:12:42.118+0000 1694f700 -1 osd.7 476 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e475) 2019-12-11T23:12:42.218 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:12:42.212+0000 1694f700 -1 osd.2 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:11:29.466828+0000 front 2019-12-11T23:11:29.637142+0000 (oldest deadline 2019-12-11T23:12:13.551894+0000) 2019-12-11T23:12:42.224 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:42.219+0000 1694f700 -1 osd.0 476 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:11:43.562919+0000 (oldest deadline 2019-12-11T23:12:23.562919+0000) 2019-12-11T23:12:42.226 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:42.222+0000 1694f700 -1 osd.0 476 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:12:43.228 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:12:43.217+0000 1694f700 -1 osd.0 477 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e474) 2019-12-11T23:13:13.414 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:13:13.411+0000 1694f700 -1 osd.1 478 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115749 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5433917920 gen 5] snapc 0=[] ondisk+write+known_if_redirected e477) 2019-12-11T23:13:55.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:13:55.274+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:13:56.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:13:56.312+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:13:57.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:13:57.319+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:13:58.305 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:13:58.299+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:13:59.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:13:59.278+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:00.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:00.267+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:01.247 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:01.240+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:02.297 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:02.292+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:02.784 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:41:49.716 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:02.794 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:41:49.725 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:02.879 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:41:49.810 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:02.887 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:41:49.818 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:02.899 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:41:49.802 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:02.902 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:41:49.804 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:03.178 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:41:50.081 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:03.267 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:41:50.199 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:03.289 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:41:50.220 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:03.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:03.295+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:04.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:04.330+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:05.160 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:41:52.092 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.175 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:41:52.106 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.184 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:41:52.115 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.193 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:41:52.124 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.208 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:41:52.110 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.208 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:41:52.139 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.222 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:41:52.124 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.230 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:41:52.161 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.294 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:41:52.196 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:05.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:05.331+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:06.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:06.316+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:07.305 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:07.299+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:08.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:08.297+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:09.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:09.253+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:10.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:10.306+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:11.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:11.352+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:12.325 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:12.320+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:13.362 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:13.357+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:14.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:14.359+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:15.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:15.404+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:16.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:16.422+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:17.476 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:17.471+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:18.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:18.481+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:19.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:19.496+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:20.514 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:20.509+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:21.484 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:21.479+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:22.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:22.468+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:23.492 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:23.487+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:24.498 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:24.493+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:25.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:25.495+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:26.456 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:26.449+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:26.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:26.516+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:26.591 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:26.586+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:27.004 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:26.997+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:27.258 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:27.254+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:27.417 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:27.411+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:27.555 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:27.551+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:27.567 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:27.560+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:27.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:27.563+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:28.029 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:28.023+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:28.222 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:28.216+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:28.398 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:28.391+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:28.553 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:28.546+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:28.553 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:28.548+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:28.567 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:28.562+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:28.935 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:28.930+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:29.003 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:28.995+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:29.203 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:29.197+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:29.443 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:29.437+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:29.540 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:29.533+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:29.541 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:29.536+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:29.565 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:29.560+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:29.962 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:29.958+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:29.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:29.969+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:30.205 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:30.201+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:30.477 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:30.471+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:30.563 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:30.559+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:30.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:30.562+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:30.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:30.564+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:30.957 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:30.949+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:31.000 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:30.995+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:31.172 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:31.165+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:31.506 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:31.500+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:31.584 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:31.579+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:31.611 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:31.603+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:31.612 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:31.607+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:31.976 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:31.970+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:31.980 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:31.972+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:32.196 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:32.192+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:32.493 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:32.488+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:32.592 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:32.585+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:32.593 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:32.587+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:32.623 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:32.619+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:32.961 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:32.956+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:32.993 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:32.988+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:33.178 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:33.175+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:33.535 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:33.529+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:33.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:33.627+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:33.634 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:33.629+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:33.672 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:33.668+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:33.960 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:33.956+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:34.030 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:34.024+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:34.180 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:34.175+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:34.525 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:34.513+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:34.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:34.643+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:34.650 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:34.645+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:34.721 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:34.717+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:35.013 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:34.986+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:35.057 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:35.051+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:35.204 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:35.199+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:35.533 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:35.528+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:35.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:35.625+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:35.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:35.628+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:35.744 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:35.741+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:36.021 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:36.018+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:36.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:36.060+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:36.206 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:36.202+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:36.522 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:36.515+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:36.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:36.624+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:36.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:36.627+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:36.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:36.749+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:37.053 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:37.048+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:37.058 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:37.052+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:37.239 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:37.235+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:37.477 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:37.471+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:37.605 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:37.598+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:37.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:37.601+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:37.801 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:37.797+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:38.071 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:38.064+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:38.084 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:38.075+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:38.203 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:38.199+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:38.452 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:38.446+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:38.651 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:38.644+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:38.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:38.647+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:38.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:38.774+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:39.067 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:39.063+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:39.084 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:39.077+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:39.243 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:39.239+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:39.487 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:39.480+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:39.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:39.648+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:39.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:39.651+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:39.792 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:39.785+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:40.048 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:40.043+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:40.133 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:40.127+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:40.215 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:40.210+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:40.489 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:40.482+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:40.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:40.617+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:40.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:40.619+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:40.835 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:40.831+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:41.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:41.061+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:41.152 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:41.145+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:41.260 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:41.256+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:41.481 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:41.475+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:41.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:41.666+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:41.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:41.669+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:41.818 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:41.814+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:42.077 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:42.071+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:42.152 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:42.145+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:42.268 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:42.263+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:42.463 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:42.457+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:42.651 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:42.645+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:42.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:42.648+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:42.794 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:42.789+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:43.105 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:43.101+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:43.132 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:43.124+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:43.267 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:43.262+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:43.492 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:43.487+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:43.698 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:43.692+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:43.699 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:43.695+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:43.815 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:43.811+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:44.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:44.140+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:44.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:44.148+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:44.318 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:44.315+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:44.505 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:44.497+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:44.661 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:44.656+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:44.662 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:44.658+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:44.849 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:44.846+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:45.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:45.104+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:45.150 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:45.146+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:45.278 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:45.273+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:45.500 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:45.495+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:45.629 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:45.622+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:45.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:45.625+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:45.881 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:45.876+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:46.077 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:46.072+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:46.118 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:46.114+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:46.332 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:46.320+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:46.478 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:46.474+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:46.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:46.577+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:46.583 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:46.579+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:46.926 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:46.921+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:47.078 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:47.073+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:47.168 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:47.164+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:47.304 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:47.300+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:47.441 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:47.435+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:47.558 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:47.552+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:47.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:47.554+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:47.960 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:47.956+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:48.102 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:48.096+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:48.169 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:48.163+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:48.281 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:48.278+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:48.401 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:48.394+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:48.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:48.547+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:48.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:48.549+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:48.927 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:48.922+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:49.089 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:49.084+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:49.131 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:49.127+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:49.238 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:49.233+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:49.439 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:49.434+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:49.596 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:49.586+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:49.596 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:49.589+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:49.910 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:49.906+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:50.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:50.108+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:50.134 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:50.129+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:50.271 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:50.266+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:50.424 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:50.418+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:50.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:50.553+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:50.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:50.556+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:50.934 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:50.930+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:51.146 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:51.140+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:51.185 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:51.179+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:51.249 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:51.246+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:51.425 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:51.418+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:51.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:51.593+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:51.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:51.596+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:51.920 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:51.908+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:52.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:14:52.125+0000 1694f700 -1 osd.2 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.568642+0000 front 2019-12-11T23:13:44.199334+0000 (oldest deadline 2019-12-11T23:14:28.860755+0000) 2019-12-11T23:14:52.234 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:14:52.228+0000 1694f700 -1 osd.4 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:42.919597+0000 front 2019-12-11T23:13:42.904658+0000 (oldest deadline 2019-12-11T23:14:26.970483+0000) 2019-12-11T23:14:52.298 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:14:52.294+0000 1694f700 -1 osd.3 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:41.082486+0000 front 2019-12-11T23:13:41.098203+0000 (oldest deadline 2019-12-11T23:14:26.856757+0000) 2019-12-11T23:14:52.438 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:14:52.431+0000 1694f700 -1 osd.6 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:44.577667+0000 front 2019-12-11T23:13:44.591678+0000 (oldest deadline 2019-12-11T23:14:26.011634+0000) 2019-12-11T23:14:52.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:52.641+0000 1694f700 -1 osd.7 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:46.144619+0000 front 2019-12-11T23:13:46.132181+0000 (oldest deadline 2019-12-11T23:14:27.357478+0000) 2019-12-11T23:14:52.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:14:52.644+0000 1694f700 -1 osd.7 484 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [getxattrs out=1026b,stat out=16b,read 0~132] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e480) 2019-12-11T23:14:52.949 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:14:52.944+0000 1694f700 -1 osd.1 484 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:13:45.588501+0000 front 2019-12-11T23:13:45.521473+0000 (oldest deadline 2019-12-11T23:14:26.429611+0000) 2019-12-11T23:14:58.049 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:42:44.980 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.079 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:42:45.011 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.100 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:42:45.032 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.109 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:42:45.040 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.138 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:42:45.041 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.144 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:42:45.047 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.162 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:42:45.094 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.185 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:42:45.116 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:14:58.270 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:42:45.172 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:15:35.726 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:35.644+0000 30f85700 -1 osd.6 488 build_incremental_map_msg missing incremental map 488 2019-12-11T23:15:35.962 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:35.779+0000 31f87700 -1 osd.6 488 build_incremental_map_msg missing incremental map 488 2019-12-11T23:15:36.080 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:35.793+0000 36790700 -1 osd.6 488 build_incremental_map_msg missing incremental map 488 2019-12-11T23:15:36.091 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:35.800+0000 34f8d700 -1 osd.6 488 build_incremental_map_msg missing incremental map 488 2019-12-11T23:15:36.133 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:35.988+0000 31786700 -1 osd.6 488 build_incremental_map_msg missing incremental map 488 2019-12-11T23:15:51.462 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:51.456+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:52.302 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:52.298+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:52.513 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:52.508+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:52.875 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:52.869+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:53.358 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:53.346+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:53.551 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:53.536+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:53.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:53.845+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:53.895 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:15:53.891+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:15:54.337 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:54.333+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:54.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:54.539+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:54.807 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:54.802+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:54.882 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:15:54.878+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:15:55.369 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:55.365+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:55.558 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:55.551+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:55.787 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:55.781+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:55.846 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:15:55.842+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:15:56.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:56.182+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:15:56.386 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:56.382+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:56.586 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:56.581+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:56.748 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:56.741+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:56.886 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:15:56.882+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:15:57.242 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:57.237+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:15:57.429 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:57.424+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:57.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:57.540+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:57.709 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:57.701+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:57.888 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:15:57.882+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:15:58.285 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:58.275+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:15:58.405 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:58.401+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:58.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:58.538+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:58.693 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:58.688+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:58.923 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:15:58.917+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:15:59.336 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:15:59.328+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:15:59.452 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:15:59.448+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:15:59.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:15:59.539+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:15:59.753 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:15:59.740+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:15:59.956 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:15:59.953+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:00.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:00.338+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:00.504 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:00.499+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:00.537 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:00.532+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:00.703 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:00.697+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:00.976 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:00.973+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:01.326 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:01.318+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:01.490 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:01.486+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:01.540 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:01.534+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:01.707 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:01.701+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:01.953 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:01.949+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:02.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:02.356+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:02.452 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:02.447+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:02.541 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:02.535+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:02.749 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:02.743+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:02.932 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:02.927+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:03.359 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:03.353+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:03.414 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:03.409+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:03.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:03.540+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:03.789 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:03.777+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:03.964 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:03.959+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:04.332 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:04.326+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:04.367 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:04.364+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:04.555 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:04.550+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:04.794 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:04.787+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:04.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:04.949+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:05.380 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:05.374+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:05.413 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:05.406+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:05.533 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:05.527+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:05.833 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:05.828+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:05.844 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:05.839+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:05.936 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:05.932+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:06.343 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:06.336+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:06.403 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:06.399+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:06.513 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:06.505+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:06.838 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:06.835+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:06.845 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:06.839+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:06.912 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:06.908+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:07.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:07.376+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:07.451 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:07.442+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:07.479 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:07.473+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:07.858 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:07.852+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:07.859 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:07.854+0000 1694f700 -1 osd.7 489 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e489) 2019-12-11T23:16:07.882 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:07.879+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:07.940 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:07.934+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:08.356 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:08.351+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:08.462 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:08.458+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:08.474 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:08.468+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:08.844 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:08.839+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:08.847 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:08.843+0000 1694f700 -1 osd.7 489 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e489) 2019-12-11T23:16:08.931 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:08.926+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:08.977 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:08.974+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:09.319 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:09.313+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:09.414 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:09.411+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:09.467 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:09.462+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:09.822 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:09.816+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:09.823 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:09.818+0000 1694f700 -1 osd.7 489 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e489) 2019-12-11T23:16:09.957 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:09.947+0000 1694f700 -1 osd.0 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:15:29.214922+0000 (oldest deadline 2019-12-11T23:16:09.214922+0000) 2019-12-11T23:16:09.957 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:09.948+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:10.000 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:09.996+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:10.311 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:10.306+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:10.375 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:10.372+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:10.510 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:10.504+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:10.795 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:10.790+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:10.796 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:10.791+0000 1694f700 -1 osd.7 489 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e489) 2019-12-11T23:16:10.983 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:10.979+0000 1694f700 -1 osd.0 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:15:29.214922+0000 (oldest deadline 2019-12-11T23:16:09.214922+0000) 2019-12-11T23:16:10.983 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:10.980+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:11.041 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:11.037+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:11.329 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:11.323+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:11.428 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:11.425+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:11.478 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:11.472+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:11.821 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:11.815+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:11.821 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:11.817+0000 1694f700 -1 osd.7 489 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e489) 2019-12-11T23:16:12.004 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:12.000+0000 1694f700 -1 osd.0 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:15:29.214922+0000 (oldest deadline 2019-12-11T23:16:09.214922+0000) 2019-12-11T23:16:12.004 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:12.001+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:12.086 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:12.082+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:12.360 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:12.355+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:12.449 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:12.445+0000 1694f700 -1 osd.1 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:07.389261+0000 front 2019-12-11T23:15:07.407647+0000 (oldest deadline 2019-12-11T23:15:52.080600+0000) 2019-12-11T23:16:12.519 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:16:12.513+0000 1694f700 -1 osd.4 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:08.195388+0000 front 2019-12-11T23:15:08.111168+0000 (oldest deadline 2019-12-11T23:15:50.934259+0000) 2019-12-11T23:16:12.789 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:12.784+0000 1694f700 -1 osd.7 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:09.951531+0000 front 2019-12-11T23:15:10.015580+0000 (oldest deadline 2019-12-11T23:15:52.845348+0000) 2019-12-11T23:16:12.792 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:12.788+0000 1694f700 -1 osd.7 489 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e489) 2019-12-11T23:16:12.985 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:12.979+0000 1694f700 -1 osd.0 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:15:29.214922+0000 (oldest deadline 2019-12-11T23:16:09.214922+0000) 2019-12-11T23:16:12.985 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:12.981+0000 1694f700 -1 osd.0 489 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:13.094 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:16:13.090+0000 1694f700 -1 osd.2 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.072749+0000 front 2019-12-11T23:15:11.087574+0000 (oldest deadline 2019-12-11T23:15:53.731882+0000) 2019-12-11T23:16:13.413 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:16:13.399+0000 1694f700 -1 osd.6 489 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:15:11.071883+0000 front 2019-12-11T23:15:11.059950+0000 (oldest deadline 2019-12-11T23:15:56.097954+0000) 2019-12-11T23:16:14.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:16:14.147+0000 1694f700 -1 osd.7 490 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:115506 2.2fs0 2.cd24ea6f (undecoded) ondisk+retry+read+known_if_redirected e489) 2019-12-11T23:16:14.280 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:16:14.242+0000 1694f700 -1 osd.0 490 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e488) 2019-12-11T23:16:44.517 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:16:44.513+0000 1694f700 -1 osd.1 491 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116085 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5433917920 gen 7] snapc 0=[] ondisk+write+known_if_redirected e490) 2019-12-11T23:17:01.706 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:17:01.556+0000 31786700 -1 osd.1 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:01.714 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:17:01.557+0000 31786700 -1 osd.1 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:01.807 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:17:01.577+0000 31786700 -1 osd.1 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:01.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:17:01.884+0000 31f87700 -1 osd.2 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:01.965 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:17:01.804+0000 35f8f700 -1 osd.1 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:01.983 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:17:01.883+0000 36790700 -1 osd.1 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.097 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:17:01.895+0000 31f87700 -1 osd.2 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.106 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:02.024+0000 35f8f700 -1 osd.6 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.169 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:17:01.932+0000 32788700 -1 osd.2 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.177 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:02.041+0000 35f8f700 -1 osd.6 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.193 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:17:01.972+0000 32788700 -1 osd.2 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.248 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:17:02.065+0000 36f91700 -1 osd.2 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:02.087+0000 31786700 -1 osd.6 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.492 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:02.271+0000 31f87700 -1 osd.6 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.526 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:02.333+0000 31f87700 -1 osd.6 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:02.529 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:02.348+0000 32788700 -1 osd.6 493 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:08.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:17:08.083+0000 3378a700 -1 osd.1 496 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:09.520 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:09.428+0000 37792700 -1 osd.6 496 build_incremental_map_msg missing incremental map 488 2019-12-11T23:17:09.622 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:17:09.540+0000 37792700 -1 osd.6 496 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:10.228 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:17:10.223+0000 32f89700 -1 osd.2 496 build_incremental_map_msg missing incremental map 493 2019-12-11T23:17:48.833 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:48.829+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:49.884 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:49.880+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:50.918 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:50.912+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:51.918 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:51.911+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:52.870 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:52.865+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:53.868 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:53.864+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:54.860 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:54.855+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:55.826 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:55.821+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:56.871 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:56.867+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:57.871 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:57.866+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:58.694 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:45:45.625 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:58.702 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:45:45.633 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:58.711 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:45:45.643 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:58.714 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:45:45.646 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:58.720 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:45:45.623 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:58.740 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:45:45.643 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:58.834 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:58.830+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:17:58.938 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:45:45.870 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:58.956 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:45:45.887 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:59.020 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:45:45.922 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:17:59.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:17:59.876+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:00.928 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:00.924+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:01.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:01.904+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:02.910 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:02.905+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:03.954 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:03.949+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:04.923 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:04.918+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:05.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:05.875+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:06.052 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:06.045+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:06.794 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:06.788+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:06.875 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:06.871+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:06.898 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:06.893+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:07.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:07.032+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:07.817 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:07.810+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:07.867 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:07.862+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:07.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:07.874+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:07.917 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:07.912+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:08.085 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:08.079+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:08.833 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:08.827+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:08.879 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:08.875+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:08.883 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:08.879+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:08.885 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:08.882+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:09.104 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:09.097+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:09.304 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:09.300+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:09.839 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:09.834+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:09.866 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:09.861+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:09.872 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:09.863+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:09.887 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:09.881+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:09.888 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:09.883+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:10.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:10.121+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:10.351 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:10.340+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:10.840 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:10.836+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:10.844 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:10.840+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:10.894 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:10.887+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:10.895 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:10.890+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:10.914 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:10.908+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:11.119 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:11.112+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:11.297 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:11.293+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:11.824 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:11.820+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:11.876 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:11.871+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:11.893 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:11.887+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:11.906 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:11.901+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:11.908 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:11.904+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:12.123 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:12.117+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:12.347 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:12.343+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:12.780 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:12.776+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:12.854 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:12.849+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:12.928 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:12.922+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:12.943 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:12.937+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:12.944 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:12.939+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:13.097 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:13.091+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:13.330 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:13.327+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:13.788 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:13.783+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:13.855 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:13.851+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:13.887 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:13.881+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:13.981 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:13.974+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:13.981 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:13.977+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:14.052 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:14.045+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:14.347 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:14.343+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:14.778 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:14.773+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:14.856 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:14.852+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:14.938 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:14.932+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:15.021 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:15.015+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:15.023 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:15.018+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:15.044 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:15.038+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:15.363 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:15.359+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:15.822 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:15.818+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:15.857 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:15.853+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:15.972 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:15.966+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:16.035 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:16.023+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:16.036 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:16.026+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:16.045 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:16.038+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:16.418 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:16.411+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:16.867 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:16.862+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:16.887 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:16.882+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:16.935 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:16.929+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:17.040 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:17.035+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:17.050 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:17.034+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:17.055 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:17.048+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:17.455 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:17.440+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:17.820 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:17.817+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:17.889 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:17.885+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:17.916 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:17.910+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:18.006 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:18.000+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:18.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:18.069+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:18.077 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:18.071+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:18.420 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:18.416+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:18.778 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:18.774+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:18.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:18.897+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:18.916 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:18.911+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:19.006 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:19.000+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:19.032 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:19.026+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:19.035 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:19.031+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:19.455 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:19.448+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:19.822 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:19.819+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:19.886 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:19.878+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:19.897 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:19.893+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:20.023 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:20.017+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:20.024 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:20.019+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:20.042 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:20.036+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:20.496 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:20.488+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:20.799 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:20.793+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:20.916 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:20.912+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:20.930 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:20.923+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:20.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:20.986+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:20.993 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:20.989+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:21.013 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:21.005+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:21.476 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:21.472+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:21.800 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:21.796+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:21.961 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:21.956+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:21.961 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:21.955+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:21.971 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:21.965+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:21.971 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:21.967+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:22.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:22.032+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:22.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:22.453+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:22.817 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:22.813+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:22.946 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:22.942+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:22.998 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:22.993+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:23.008 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:23.002+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:23.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:23.014+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:23.022 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:23.018+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:23.464 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:23.457+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:23.871 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:23.865+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:23.975 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:23.970+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:23.978 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:23.973+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:24.010 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:24.005+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:24.011 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:24.008+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:24.047 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:24.040+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:24.499 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:24.495+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:24.921 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:24.917+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:24.971 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:24.964+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:24.976 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:24.972+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:25.050 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:25.045+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:25.053 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:25.049+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:25.072 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:25.060+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:25.471 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:25.466+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:25.935 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:25.928+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:25.974 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:25.969+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:26.043 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:26.026+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:26.058 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:26.023+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:26.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:26.024+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:26.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:26.031+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:26.454 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:26.450+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:26.953 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:26.947+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:26.989 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:26.984+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:27.024 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:27.018+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:27.027 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:27.023+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:27.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:27.031+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:27.055 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:27.051+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:27.433 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:27.428+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:27.915 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:27.909+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:28.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:28.011+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:28.032 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:28.029+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:28.074 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:28.068+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:28.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:28.071+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:28.086 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:28.080+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:28.466 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:28.461+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:28.898 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:28.892+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:28.989 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:28.984+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:29.064 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:29.058+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:29.065 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:29.060+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:29.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:29.062+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:29.070 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:29.065+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:29.514 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:29.509+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:29.945 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:29.940+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:29.980 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:29.973+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:30.034 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:30.031+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:30.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:30.061+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:30.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:30.095+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:30.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:30.098+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:30.554 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:30.549+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:30.968 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:30.962+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:31.009 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:30.994+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:31.030 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:31.023+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:31.071 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:31.062+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:31.128 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:31.122+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:31.131 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:31.127+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:31.565 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:31.559+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:31.942 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:31.935+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:31.996 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:31.990+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:32.036 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:32.031+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:32.046 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:32.039+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:32.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:32.110+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:32.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:32.112+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:32.540 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:32.536+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:32.910 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:32.905+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:33.006 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:33.000+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:33.019 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:33.012+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:33.084 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:33.080+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:33.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:33.157+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:33.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:33.158+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:33.547 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:33.542+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:33.947 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:33.942+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:34.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:34.032+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:34.069 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:34.064+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:34.134 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:34.129+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:34.206 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:34.201+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:34.210 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:34.206+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:34.558 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:34.553+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:34.915 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:34.910+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:35.053 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:35.047+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:35.106 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:35.100+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:35.145 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:35.140+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:35.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:35.186+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:35.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:35.191+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:35.517 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:35.513+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:35.893 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:35.887+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:36.101 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:36.095+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:36.115 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:36.108+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:36.190 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:36.185+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:36.237 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:36.230+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:36.237 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:36.232+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:36.534 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:36.528+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:36.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:36.843+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:37.080 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:37.074+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:37.141 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:37.136+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:37.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:37.188+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:37.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:37.191+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:37.234 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:37.230+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:37.578 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:37.572+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:37.883 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:18:37.878+0000 1694f700 -1 osd.3 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:22.728265+0000 front 2019-12-11T23:17:22.814883+0000 (oldest deadline 2019-12-11T23:18:06.594229+0000) 2019-12-11T23:18:38.077 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:18:38.073+0000 1694f700 -1 osd.6 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.985776+0000 front 2019-12-11T23:17:25.924539+0000 (oldest deadline 2019-12-11T23:18:06.405449+0000) 2019-12-11T23:18:38.166 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:38.161+0000 1694f700 -1 osd.7 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:25.090261+0000 front 2019-12-11T23:17:25.111047+0000 (oldest deadline 2019-12-11T23:18:09.170995+0000) 2019-12-11T23:18:38.167 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:38.164+0000 1694f700 -1 osd.7 498 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:38.183 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:18:38.178+0000 1694f700 -1 osd.4 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:20.089439+0000 front 2019-12-11T23:17:26.017226+0000 (oldest deadline 2019-12-11T23:18:05.965942+0000) 2019-12-11T23:18:38.218 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:18:38.212+0000 1694f700 -1 osd.2 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.610645+0000 front 2019-12-11T23:17:24.370047+0000 (oldest deadline 2019-12-11T23:18:07.839312+0000) 2019-12-11T23:18:38.568 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:18:38.563+0000 1694f700 -1 osd.1 498 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:17:24.362259+0000 front 2019-12-11T23:17:24.498041+0000 (oldest deadline 2019-12-11T23:18:08.367503+0000) 2019-12-11T23:18:38.760 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:18:38.751+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 41sec e498 v498) 2019-12-11T23:18:39.835 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:39.708+0000 1694f700 -1 osd.7 499 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:40.694 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:40.690+0000 1694f700 -1 osd.7 500 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:18:41.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:18:41.888+0000 1694f700 -1 osd.7 500 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116227 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs out=940b,stat out=16b,read 0~128] snapc 0=[] ondisk+read+known_if_redirected e498) 2019-12-11T23:19:09.039 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:09.035+0000 1694f700 -1 osd.1 500 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116359 5.5 5:a93a5511:::notify.2:head [watch reconnect cookie 821300688 gen 133] snapc 0=[] ondisk+write+known_if_redirected e499) 2019-12-11T23:19:11.673 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:11.659+0000 21452700 -1 reset not still connected to 0xec01360 2019-12-11T23:19:11.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:11.660+0000 21452700 -1 reset not still connected to 0x100ccda0 2019-12-11T23:19:36.417 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:36.413+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:36.904 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:36.900+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:37.395 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:37.390+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:37.871 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:37.868+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:38.401 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:38.396+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:38.922 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:38.919+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:39.022 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:39.012+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:39.095 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:39.090+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:39.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:39.383+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:39.885 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:39.880+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:40.002 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:39.998+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:40.069 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:40.066+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:40.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:40.424+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:40.888 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:40.882+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:40.965 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:40.961+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:41.069 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:41.064+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:41.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:41.402+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:41.893 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:41.889+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:42.014 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:42.009+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:42.090 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:42.080+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:42.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:42.445+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:42.931 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:42.928+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:42.997 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:42.993+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:43.121 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:43.117+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:43.434 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:43.428+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:43.970 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:43.964+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:44.016 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:44.010+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:44.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:44.129+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:44.431 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:44.427+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:44.949 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:44.939+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:45.068 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:45.063+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:45.129 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:45.116+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:45.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:45.439+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:45.977 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:45.974+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:46.075 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:46.071+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:46.100 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:46.095+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:46.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:46.420+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:46.989 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:46.985+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:47.045 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:47.041+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:47.063 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:47.059+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:47.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:47.149+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:47.443 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:47.438+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:48.016 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:48.008+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:48.022 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:48.019+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:48.039 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:48.035+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:48.184 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:48.177+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:48.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:48.432+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:48.999 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:48.996+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:49.012 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:49.009+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:49.029 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:49.026+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:49.228 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:49.223+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:49.410 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:49.404+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:50.016 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:50.011+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:50.029 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:50.024+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:50.054 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:50.051+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:50.247 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:50.243+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:50.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:50.435+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:50.981 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:50.978+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:51.061 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:51.057+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:51.076 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:51.072+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:51.299 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:51.294+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:51.433 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:51.427+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:51.957 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:51.953+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:52.044 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:52.040+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:52.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:52.061+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:52.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:52.299+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:52.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:52.397+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:52.982 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:52.979+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:53.009 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:53.006+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:53.059 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:53.056+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:53.354 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:53.349+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:53.415 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:53.410+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:54.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:54.014+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:54.024 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:54.020+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:19:54.024 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:54.021+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:54.050 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:54.047+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:54.377 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:54.372+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:54.442 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:54.436+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:54.984 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:54.980+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:55.047 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:55.044+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:55.060 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:55.056+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:19:55.060 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:55.057+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:55.357 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:55.352+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:55.492 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:55.486+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:55.947 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:55.943+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:56.003 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:56.000+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:56.071 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:56.067+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:19:56.072 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:56.069+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:56.346 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:56.340+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:56.481 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:56.476+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:56.917 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:56.914+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:57.027 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:57.023+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:57.072 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:57.066+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:19:57.076 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:57.069+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:57.308 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:57.303+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:57.467 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:57.463+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:57.904 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:57.901+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:57.986 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:57.982+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:58.061 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:58.057+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:19:58.063 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:58.059+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:58.353 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:58.348+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:58.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:58.464+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:58.863 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:58.860+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:58.966 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:58.963+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:19:59.072 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:59.069+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:19:59.073 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:19:59.071+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:19:59.310 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:19:59.304+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:19:59.491 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:19:59.486+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:19:59.828 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:19:59.824+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:19:59.938 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:19:59.935+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:00.040 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:00.036+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:00.042 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:00.040+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:00.329 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:00.325+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:00.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:00.501+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:00.804 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:00.800+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:00.957 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:00.953+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:01.060 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:01.057+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:01.062 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:01.060+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:01.331 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:01.324+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:01.531 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:01.525+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:01.846 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:01.841+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:01.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:01.930+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:02.108 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:02.102+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:02.108 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:02.104+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:02.367 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:02.359+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:02.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:02.576+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:02.852 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:02.848+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:02.955 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:02.949+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:03.126 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:03.120+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:03.127 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:03.123+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:03.334 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:03.330+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:03.602 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:03.597+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:03.810 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:03.805+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:03.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:03.968+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:04.162 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:04.156+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:04.164 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:04.160+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:04.368 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:04.364+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:04.605 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:04.599+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:04.766 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:04.762+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:05.020 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:05.015+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:05.152 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:05.148+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:05.154 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:05.150+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:05.395 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:05.391+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:05.613 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:05.609+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:05.728 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:05.722+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:06.009 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:06.004+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:06.186 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:06.180+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:06.190 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:06.183+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:06.395 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:06.390+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:06.654 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:06.649+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:06.734 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:06.724+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:07.026 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:07.021+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:07.237 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:07.231+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:07.237 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:07.233+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:07.367 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:07.362+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:07.690 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:07.685+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:07.691 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:07.686+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:08.058 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:08.055+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:08.241 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:08.233+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:08.243 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:08.235+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:08.421 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:08.415+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:08.676 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:08.670+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:08.714 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:08.711+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:09.009 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:20:08.996+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e503 v503) 2019-12-11T23:20:09.103 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:09.100+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:09.231 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:09.226+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:09.231 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:09.228+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:09.475 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:09.470+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:09.707 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:09.697+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:09.716 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:09.712+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:10.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:10.085+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:10.255 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:10.252+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:10.255 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:10.253+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:10.522 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:10.516+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:10.695 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:10.690+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:10.761 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:10.757+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:11.112 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:11.107+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:11.264 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:11.259+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:11.264 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:11.261+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:11.491 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:11.484+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:11.503 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:11.497+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:11.677 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:11.672+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:11.757 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:11.752+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:12.131 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:12.127+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:12.236 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:12.229+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:12.236 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:12.231+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:12.479 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:12.474+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:12.523 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:12.518+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:12.672 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:12.667+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:12.807 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:12.804+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:13.091 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:13.088+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:13.251 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:13.248+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:13.253 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:13.250+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:13.528 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:13.523+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:13.533 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:13.528+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:13.689 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:13.684+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:13.787 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:13.783+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:14.017 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:20:14.014+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e503 v503) 2019-12-11T23:20:14.077 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:14.073+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:14.272 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:14.268+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:14.273 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:14.269+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:14.529 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:14.521+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:14.535 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:14.524+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:14.718 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:14.713+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:14.784 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:14.780+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:15.071 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:15.067+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:15.241 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:15.236+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:15.241 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:15.238+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:15.489 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:15.483+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:15.495 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:15.490+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:15.744 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:15.737+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:15.826 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:15.822+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:16.031 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:16.027+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:16.220 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:16.213+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:16.220 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:16.215+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:16.499 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:16.494+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:16.539 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:16.534+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:16.706 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:16.701+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:16.839 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:16.835+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:17.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:17.043+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:17.251 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:17.245+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:17.251 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:17.246+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:17.525 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:17.516+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:17.533 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:17.525+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:17.703 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:17.698+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:17.793 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:17.790+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:18.088 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:18.083+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:18.246 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:18.243+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:18.248 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:18.246+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:18.509 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:18.503+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:18.572 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:18.567+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:18.682 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:18.677+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:18.777 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:18.773+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:19.033 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:20:19.030+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e503 v503) 2019-12-11T23:20:19.059 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:19.055+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:19.254 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:19.251+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:19.255 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:19.253+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:19.555 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:19.550+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:19.615 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:19.610+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:19.675 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:19.670+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:19.799 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:19.795+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:20.055 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:20.051+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:20.298 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:20.293+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:20.298 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:20.295+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:20.560 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:20.554+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:20.595 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:20.591+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:20.685 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:20.679+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:20.760 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:20.757+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:21.018 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:21.014+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:21.284 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:21.280+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:21.285 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:21.282+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:21.564 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:21.558+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:21.639 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:21.634+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:21.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:21.637+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:21.736 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:21.733+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:22.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:22.040+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:22.246 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:22.242+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:22.247 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:22.243+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:22.561 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:22.554+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:22.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:22.625+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:22.666 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:22.661+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:22.752 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:22.748+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:23.033 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:23.028+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:23.217 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:23.214+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:23.219 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:23.215+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:23.521 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:23.513+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:23.644 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:23.638+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:23.698 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:23.692+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:23.710 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:23.707+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:24.060 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:20:24.054+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e503 v503) 2019-12-11T23:20:24.069 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:24.066+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:24.180 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:24.177+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:24.181 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:24.178+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:24.528 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:24.524+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:24.689 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:24.684+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:24.725 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:24.719+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:24.753 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:24.748+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:25.039 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:25.035+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:25.224 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:25.221+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:25.225 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:25.222+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:25.503 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:25.498+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:25.701 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:25.696+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:25.742 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:25.737+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:25.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:25.795+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:26.057 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:26.053+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:26.207 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:26.203+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:26.209 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:26.206+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:26.484 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:26.479+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:26.705 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:26.699+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:26.764 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:26.757+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:26.782 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:26.778+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:27.107 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:27.103+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:27.202 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:27.198+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:27.203 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:27.199+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:27.531 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:27.526+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:27.689 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:27.684+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:27.771 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:27.767+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:27.796 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:27.790+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:28.101 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:28.097+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:28.185 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:28.181+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:28.188 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:28.185+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:20:28.587 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:20:28.582+0000 1694f700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.391943+0000 front 2019-12-11T23:18:54.984600+0000 (oldest deadline 2019-12-11T23:20:10.983120+0000) 2019-12-11T23:20:28.729 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:20:28.724+0000 1694f700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817952+0000 front 2019-12-11T23:18:54.819906+0000 (oldest deadline 2019-12-11T23:19:38.083092+0000) 2019-12-11T23:20:28.795 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:20:28.790+0000 1694f700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:55.395552+0000 front 2019-12-11T23:18:55.096154+0000 (oldest deadline 2019-12-11T23:19:36.154227+0000) 2019-12-11T23:20:28.811 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:20:28.807+0000 1694f700 -1 osd.2 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:54.817279+0000 front 2019-12-11T23:18:54.714507+0000 (oldest deadline 2019-12-11T23:19:36.273863+0000) 2019-12-11T23:20:29.069 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:20:29.064+0000 1694f700 -1 osd.1 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:18:53.333556+0000 front 2019-12-11T23:18:53.335385+0000 (oldest deadline 2019-12-11T23:19:38.441330+0000) 2019-12-11T23:20:29.122 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:20:29.117+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e503 v503) 2019-12-11T23:20:29.212 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:29.193+0000 1694f700 -1 osd.0 503 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:19:13.504498+0000 (oldest deadline 2019-12-11T23:19:53.504498+0000) 2019-12-11T23:20:29.212 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:20:29.195+0000 1694f700 -1 osd.0 503 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e502) 2019-12-11T23:21:28.515 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:28.511+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:29.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:29.469+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:30.460 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:30.456+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:31.493 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:31.489+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:32.456 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:32.453+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:33.424 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:33.421+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:34.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:34.472+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:35.529 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:35.519+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:36.496 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:36.493+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:37.516 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:37.511+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:38.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:38.504+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:39.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:39.472+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:40.520 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:40.516+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:41.543 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:41.538+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:42.518 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:42.513+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:43.497 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:43.493+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:44.525 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:44.521+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:45.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:45.558+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:46.580 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:46.576+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:47.547 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:47.542+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:48.524 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:48.520+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:49.556 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:49.551+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:50.537 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:50.533+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:51.587 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:51.584+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:52.585 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:52.580+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:53.595 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:53.591+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:54.467 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:21:54.460+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:21:54.566 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:54.562+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:54.584 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:21:54.581+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:21:54.587 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:49:41.519 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.594 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:49:41.526 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.607 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:49:41.539 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.611 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:49:41.514 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.613 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:49:41.545 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.622 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:49:41.525 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.872 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:49:41.804 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.876 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:49:41.808 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:54.946 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:49:41.849 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:21:55.440 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:21:55.435+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:21:55.547 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:55.542+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:21:55.548 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:55.545+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:55.597 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:21:55.593+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:21:56.433 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:21:56.428+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:21:56.514 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:56.510+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:21:56.515 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:56.511+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:56.634 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:21:56.631+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:21:56.903 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:21:56.900+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:21:57.127 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:21:57.123+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:21:57.253 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:21:57.249+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:21:57.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:21:57.401+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:21:57.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:57.479+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:21:57.487 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:57.482+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:57.661 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:21:57.657+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:21:57.871 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:21:57.868+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:21:58.124 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:21:58.121+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:21:58.232 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:21:58.227+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:21:58.425 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:21:58.417+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:21:58.452 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:58.446+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:21:58.453 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:58.449+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:58.630 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:21:58.626+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:21:58.881 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:21:58.876+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:21:59.080 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:21:59.075+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:21:59.212 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:21:59.205+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:21:59.442 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:21:59.437+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:21:59.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:59.465+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:21:59.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:21:59.469+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:21:59.666 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:21:59.663+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:21:59.923 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:21:59.920+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:00.097 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:00.094+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:00.256 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:00.250+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:00.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:00.441+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:00.448 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:00.443+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:00.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:00.445+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:00.678 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:00.674+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:00.958 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:00.955+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:01.079 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:01.076+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:01.250 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:01.244+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:01.417 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:01.412+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:01.418 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:01.414+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:01.456 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:01.450+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:01.687 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:01.683+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:01.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:01.982+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:02.101 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:02.097+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:02.223 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:02.215+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:02.434 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:02.428+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:02.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:02.417+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:02.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:02.420+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:02.690 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:02.686+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:02.974 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:02.970+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:03.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:03.082+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:03.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:03.177+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:03.418 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:03.412+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:03.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:03.415+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:03.448 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:03.444+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:03.698 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:03.693+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:04.016 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:04.011+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:04.131 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:04.127+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:04.186 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:04.180+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:04.387 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:04.381+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:04.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:04.384+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:04.413 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:04.407+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:04.693 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:04.689+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:04.973 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:04.969+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:05.157 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:05.153+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:05.202 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:05.196+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:05.373 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:05.367+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:05.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:05.435+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:05.442 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:05.438+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:05.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:05.650+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:05.956 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:05.952+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:06.126 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:06.122+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:06.230 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:06.224+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:06.375 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:06.369+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:06.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:06.464+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:06.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:06.467+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:06.659 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:06.655+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:07.004 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:07.000+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:07.141 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:07.137+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:07.197 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:07.191+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:07.363 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:07.357+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:07.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:07.422+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:07.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:07.425+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:07.617 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:07.613+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:07.993 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:07.990+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:08.102 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:08.098+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:08.202 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:08.197+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:08.402 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:08.396+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:08.437 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:08.430+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:08.438 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:08.433+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:08.631 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:08.628+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:08.959 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:08.955+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:09.076 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:09.073+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:09.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:09.169+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:09.404 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:09.399+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:09.466 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:09.460+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:09.466 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:09.462+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:09.622 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:09.618+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:09.944 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:09.939+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:10.036 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:10.031+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:10.225 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:10.218+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:10.374 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:10.368+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:10.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:10.449+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:10.456 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:10.452+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:10.664 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:10.660+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:10.945 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:10.941+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:11.063 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:11.059+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:11.219 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:11.213+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:11.389 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:11.383+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:11.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:11.406+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:11.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:11.410+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:11.626 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:11.622+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:11.916 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:11.911+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:12.050 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:12.046+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:12.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:12.258+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:12.393 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:12.384+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:12.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:12.386+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:12.400 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:12.395+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:12.652 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:12.649+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:12.888 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:12.885+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:13.035 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:13.031+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:13.283 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:13.279+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:13.341 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:13.337+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:13.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:13.338+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:13.352 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:13.348+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:13.690 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:13.684+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:13.926 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:13.922+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:14.046 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:14.041+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:14.280 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:14.275+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:14.310 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:14.305+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:14.382 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:14.377+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:14.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:14.381+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:14.665 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:14.660+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:14.922 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:14.917+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:15.031 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:15.026+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:15.268 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:15.262+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:15.306 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:15.301+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:15.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:15.383+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:15.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:15.386+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:15.710 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:15.704+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:15.968 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:15.964+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:16.014 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:16.010+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:16.288 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:16.283+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:16.297 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:16.291+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:16.379 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:16.373+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:16.380 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:16.375+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:16.691 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:16.687+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:17.005 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:16.999+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:17.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:17.044+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:17.284 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:17.279+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:17.286 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:17.281+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:17.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:17.413+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:17.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:17.415+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:17.729 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:17.723+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:18.041 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:18.036+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:18.052 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:18.047+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:18.311 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:18.306+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:18.325 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:18.320+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:18.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:18.381+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:18.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:18.383+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:18.713 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:18.708+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:18.998 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:18.994+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:19.007 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:19.003+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:19.336 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:19.331+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:19.357 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:19.353+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:19.362 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:19.355+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:19.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:19.361+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:19.742 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:19.733+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:19.955 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:19.951+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:20.009 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:20.005+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:20.325 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:20.319+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:20.325 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:20.321+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:20.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:20.329+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:20.340 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:20.334+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:20.718 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:20.714+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:20.944 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:20.940+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:21.037 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:21.033+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:21.325 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:21.319+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:21.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:21.322+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:21.345 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:21.339+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:21.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:21.344+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:21.685 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:21.682+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:21.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:21.969+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:22.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:22.013+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:22.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:22.304+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:22.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:22.307+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:22.325 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:22.319+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:22.377 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:22.371+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:22.642 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:22.637+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:22.975 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:22.971+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:22.990 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:22.986+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:23.275 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:23.269+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:23.276 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:23.272+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:23.370 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:23.364+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:23.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:23.401+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:23.673 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:23.666+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:23.978 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:23.973+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:24.018 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:24.014+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:24.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:24.259+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:24.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:24.262+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:24.371 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:24.364+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:24.397 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:24.392+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:24.566 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:22:24.563+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 45sec e509 v509) 2019-12-11T23:22:24.651 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:24.648+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:24.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:24.938+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:25.059 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:25.055+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:25.309 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:25.303+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:25.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:25.306+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:25.372 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:25.367+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:25.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:25.366+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:25.665 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:25.662+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:25.967 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:25.963+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:26.013 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:26.009+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:26.282 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:26.276+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:26.361 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:26.281+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:26.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:26.340+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:26.381 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:26.375+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:26.656 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:26.651+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:26.966 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:26.960+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:26.980 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:26.976+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:27.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:27.311+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:27.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:27.314+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:27.326 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:27.320+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:27.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:27.401+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:27.673 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:27.668+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:27.920 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:27.916+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:28.029 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:28.022+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:28.342 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:28.337+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:28.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:28.347+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:28.355 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:28.351+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:28.387 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:28.382+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:28.704 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:22:28.698+0000 1694f700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.921226+0000 front 2019-12-11T23:21:10.004278+0000 (oldest deadline 2019-12-11T23:21:53.876442+0000) 2019-12-11T23:22:28.908 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:22:28.905+0000 1694f700 -1 osd.2 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.387863+0000 front 2019-12-11T23:21:13.350890+0000 (oldest deadline 2019-12-11T23:21:56.227277+0000) 2019-12-11T23:22:28.984 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:22:28.981+0000 1694f700 -1 osd.1 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.431498+0000 front 2019-12-11T23:21:10.502007+0000 (oldest deadline 2019-12-11T23:21:56.317395+0000) 2019-12-11T23:22:29.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:29.345+0000 1694f700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:10.196791+0000 front 2019-12-11T23:21:10.210875+0000 (oldest deadline 2019-12-11T23:21:54.606209+0000) 2019-12-11T23:22:29.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:29.347+0000 1694f700 -1 osd.7 509 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:29.368 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:22:29.363+0000 1694f700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:13.987480+0000 front 2019-12-11T23:21:14.024059+0000 (oldest deadline 2019-12-11T23:21:56.804199+0000) 2019-12-11T23:22:29.409 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:22:29.404+0000 1694f700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:21:08.744624+0000 front 2019-12-11T23:21:08.728732+0000 (oldest deadline 2019-12-11T23:21:53.903827+0000) 2019-12-11T23:22:29.623 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:22:29.616+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 45sec e509 v509) 2019-12-11T23:22:30.774 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:30.771+0000 1694f700 -1 osd.7 510 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:31.861 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:31.829+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:32.867 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:32.862+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:33.828 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:33.825+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:34.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:34.806+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:35.764 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:35.761+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:36.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:36.740+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:37.767 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:37.764+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:38.757 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:38.751+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:39.792 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:39.789+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:40.751 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:40.748+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:41.755 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:41.750+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:42.783 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:42.779+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:43.835 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:43.831+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:44.797 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:44.793+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:45.760 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:45.756+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:46.756 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:46.753+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:47.804 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:47.800+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:48.786 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:48.782+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:49.756 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:49.748+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:50.786 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:50.783+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:51.840 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:51.836+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:52.844 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:52.840+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:53.835 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:53.830+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:54.843 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:54.839+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:55.924 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:55.915+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:56.885 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:56.881+0000 1694f700 -1 osd.7 511 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:57.859 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:57.855+0000 1694f700 -1 osd.7 512 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:22:58.871 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:22:58.867+0000 1694f700 -1 osd.7 512 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:00.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:00.304+0000 1694f700 -1 osd.7 513 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:01.322 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:01.319+0000 1694f700 -1 osd.7 514 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:02.395 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:02.299+0000 1694f700 -1 osd.7 515 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:03.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:03.258+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:04.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:04.340+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:05.379 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:05.376+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:06.418 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:06.415+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:07.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:07.407+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:08.464 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:08.460+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:09.477 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:09.473+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:10.531 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:10.526+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:11.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:11.508+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:12.555 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:12.551+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:13.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:13.572+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:14.532 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:14.528+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:15.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:15.580+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:16.547 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:16.542+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:17.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:17.594+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:18.595 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:18.591+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:19.553 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:19.549+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:19.917 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:51:06.849 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:19.948 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:51:06.880 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:19.949 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:51:06.881 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:19.966 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:51:06.898 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:19.991 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:51:06.894 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:19.997 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:51:06.900 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:20.060 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:51:06.992 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:20.082 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:51:07.014 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:20.154 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:51:07.057 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:23:20.578 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:20.574+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:21.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:21.593+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:22.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:22.559+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:23.549 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:23.545+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:24.529 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:24.524+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:25.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:25.572+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:26.612 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:26.607+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:27.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:27.568+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:28.571 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:28.567+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:29.219 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:29.216+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:29.593 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:29.589+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:29.612 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:29.608+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:30.098 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:30.095+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:30.189 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:30.184+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:30.566 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:30.562+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:30.658 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:30.654+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:30.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:30.656+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:31.111 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:31.108+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:31.225 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:31.220+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:31.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:31.347+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:31.601 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:31.597+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:31.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:31.643+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:31.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:31.645+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:32.014 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:32.010+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:32.113 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:32.110+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:32.271 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:32.268+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:32.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:32.308+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:32.600 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:32.596+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:32.647 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:32.642+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:32.647 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:32.644+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:33.037 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:33.033+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:33.144 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:33.141+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:33.286 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:33.282+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:33.316 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:33.312+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:33.593 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:33.580+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:33.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:33.613+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:33.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:33.615+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:34.038 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:34.035+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:34.121 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:34.116+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:34.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:34.279+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:34.295 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:34.289+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:34.627 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:34.623+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:34.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:34.624+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:34.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:34.627+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:35.002 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:34.997+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:35.106 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:35.095+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:35.291 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:35.287+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:35.306 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:35.303+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:35.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:35.607+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:35.614 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:35.611+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:35.658 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:35.655+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:36.027 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:36.022+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:36.144 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:36.141+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:36.283 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:36.276+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:36.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:36.299+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:36.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:36.577+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:36.585 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:36.579+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:36.665 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:36.661+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:37.061 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:37.054+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:37.191 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:37.188+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:37.297 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:37.292+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:37.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:37.309+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:37.583 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:37.577+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:37.583 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:37.579+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:37.702 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:37.696+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:38.020 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:38.015+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:38.163 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:38.160+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:38.165 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:38.162+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:38.297 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:38.291+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:38.327 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:38.324+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:38.620 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:38.616+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:38.622 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:38.619+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:38.714 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:38.711+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:39.049 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:39.044+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:39.198 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:39.193+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:39.204 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:39.196+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:39.273 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:39.268+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:39.330 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:39.327+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:39.656 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:39.650+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:39.658 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:39.652+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:39.701 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:39.697+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:40.104 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:40.098+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:40.237 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:40.232+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:40.238 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:40.235+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:40.283 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:40.278+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:40.363 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:40.354+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:40.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:40.613+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:40.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:40.615+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:40.685 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:40.674+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:41.135 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:41.130+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:41.248 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:41.243+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:41.249 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:41.244+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:41.270 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:41.265+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:41.367 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:41.362+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:41.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:41.612+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:41.620 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:41.614+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:41.716 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:41.709+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:42.111 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:42.106+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:42.249 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:42.244+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:42.293 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:42.288+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:42.295 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:42.291+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:42.349 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:42.344+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:42.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:42.620+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:42.628 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:42.624+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:42.714 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:42.709+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:43.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:43.131+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:43.207 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:43.201+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:43.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:43.332+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:43.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:43.333+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:43.339 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:43.335+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:43.677 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:43.671+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:43.677 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:43.673+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:43.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:43.748+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:44.166 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:44.160+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:44.234 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:44.229+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:44.319 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:44.313+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:44.356 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:44.350+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:44.357 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:44.353+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:44.667 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:44.662+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:44.668 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:44.664+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:44.763 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:44.759+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:45.145 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:45.140+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:45.248 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:45.243+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:45.293 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:45.286+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:45.370 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:45.366+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:45.371 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:45.367+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:45.700 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:45.694+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:45.700 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:45.696+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:45.744 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:45.739+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:46.125 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:46.119+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:46.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:46.197+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:46.273 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:46.269+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:46.332 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:46.326+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:46.333 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:46.328+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:46.741 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:46.736+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:46.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:46.740+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:46.787 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:46.782+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:47.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:47.111+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:47.230 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:47.225+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:47.273 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:47.269+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:47.364 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:47.359+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:47.364 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:47.360+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:47.770 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:47.767+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:47.789 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:47.785+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:47.792 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:47.789+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:48.076 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:48.070+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:48.248 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:48.244+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:48.265 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:48.260+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:48.352 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:48.347+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:48.353 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:48.348+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:48.751 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:48.748+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:48.756 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:48.751+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:48.757 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:48.753+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:49.106 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:49.101+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:49.230 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:49.227+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:49.288 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:49.283+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:49.368 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:49.364+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:49.368 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:49.365+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:49.801 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:49.797+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:49.810 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:49.805+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:49.811 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:49.806+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:50.103 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:50.099+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:50.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:50.229+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:50.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:50.298+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:50.415 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:50.412+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:50.417 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:50.414+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:50.785 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:50.781+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:50.818 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:50.812+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:50.819 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:50.814+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:51.068 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:51.063+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:51.191 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:51.188+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:51.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:51.299+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:51.379 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:51.376+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:51.381 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:51.379+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:51.759 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:51.754+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:51.860 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:51.855+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:51.863 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:51.859+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:52.108 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:52.097+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:52.180 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:52.176+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:52.302 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:52.297+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:52.357 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:52.353+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:52.359 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:52.357+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:52.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:52.750+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:52.875 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:52.868+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:52.878 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:52.874+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:53.128 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:53.122+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:53.188 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:53.185+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:53.261 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:53.255+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:53.365 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:53.352+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:53.365 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:53.353+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:53.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:53.774+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:53.919 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:53.914+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:53.922 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:53.918+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:54.165 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:23:54.160+0000 1694f700 -1 osd.6 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.775429+0000 front 2019-12-11T23:22:46.912837+0000 (oldest deadline 2019-12-11T23:23:31.265294+0000) 2019-12-11T23:23:54.214 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:54.210+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:54.308 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:54.303+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:54.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:54.403+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:54.407 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:54.404+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:54.797 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:54.794+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:54.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:54.904+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:54.910 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:54.906+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:55.232 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:55.229+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:55.298 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:55.294+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:55.363 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:55.358+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:55.363 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:55.360+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:55.796 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:55.792+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:55.875 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:55.870+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:55.876 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:55.872+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:56.239 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:56.234+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:56.256 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:56.251+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:56.411 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:56.407+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:56.419 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:56.408+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:56.789 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:56.785+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:56.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:56.846+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:56.852 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:56.848+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:57.239 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:57.235+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:57.243 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:57.238+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:57.407 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:57.404+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:57.410 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:57.407+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:57.827 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:57.823+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:57.832 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:57.826+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:57.832 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:57.828+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:58.226 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:58.222+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:58.266 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:58.263+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:58.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:58.395+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:58.400 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:58.397+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:58.798 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:58.793+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:58.801 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:58.797+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:58.835 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:58.831+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:23:59.230 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:23:59.225+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:23:59.309 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:23:59.305+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:23:59.411 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:59.407+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:23:59.412 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:23:59.409+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:23:59.833 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:59.829+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:23:59.836 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:23:59.833+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:23:59.884 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:23:59.881+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:00.205 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:00.200+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:00.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:00.292+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:00.396 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:00.393+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:00.398 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:00.396+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:00.853 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:00.848+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:00.854 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:00.850+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:00.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:00.930+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:01.258 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:01.253+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:01.302 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:01.298+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:01.426 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:01.422+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:01.427 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:01.424+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:01.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:01.876+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:01.882 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:01.878+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:01.949 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:01.945+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:02.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:02.219+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:02.330 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:02.327+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:02.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:02.430+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:02.434 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:02.431+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:02.930 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:02.926+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:02.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:02.930+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:02.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:02.990+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:03.242 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:03.236+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:03.304 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:03.300+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:03.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:03.394+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:03.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:03.398+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:03.929 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:03.924+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:03.930 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:03.926+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:03.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:03.969+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:04.233 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:04.227+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:04.337 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:04.327+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:04.373 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:04.370+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:04.376 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:04.373+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:04.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:04.906+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:04.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:04.907+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:04.975 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:04.971+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:05.011 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:24:05.008+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e516 v516) 2019-12-11T23:24:05.262 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:05.255+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:05.287 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:05.283+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:05.381 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:05.378+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:05.383 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:05.380+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:05.947 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:05.924+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:05.949 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:05.927+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:05.980 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:05.976+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:06.236 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:06.231+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:06.267 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:06.264+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:06.410 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:06.407+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:06.411 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:06.409+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:06.961 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:06.955+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:06.962 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:06.957+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:07.022 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:07.019+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:07.231 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:07.226+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:07.285 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:07.282+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:07.411 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:07.403+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:07.411 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:07.405+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:07.923 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:07.916+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:07.924 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:07.917+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:08.031 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:08.027+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:08.190 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:08.185+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:08.307 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:08.302+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:08.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:08.367+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:08.375 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:08.372+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:08.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:08.924+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:08.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:08.926+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:09.056 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:09.052+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:09.233 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:24:09.227+0000 1694f700 -1 osd.4 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.984815+0000 front 2019-12-11T23:22:46.927788+0000 (oldest deadline 2019-12-11T23:23:28.283103+0000) 2019-12-11T23:24:09.295 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:24:09.291+0000 1694f700 -1 osd.2 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.782498+0000 front 2019-12-11T23:22:45.608408+0000 (oldest deadline 2019-12-11T23:23:30.906310+0000) 2019-12-11T23:24:09.369 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:09.363+0000 1694f700 -1 osd.0 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:22:57.725804+0000 (oldest deadline 2019-12-11T23:23:37.725804+0000) 2019-12-11T23:24:09.370 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:09.365+0000 1694f700 -1 osd.0 516 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:09.887 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:09.881+0000 1694f700 -1 osd.7 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:45.474898+0000 front 2019-12-11T23:22:45.238885+0000 (oldest deadline 2019-12-11T23:23:30.490757+0000) 2019-12-11T23:24:09.887 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:09.883+0000 1694f700 -1 osd.7 516 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:10.057 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:10.054+0000 1694f700 -1 osd.1 516 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:22:46.987691+0000 front 2019-12-11T23:22:46.914371+0000 (oldest deadline 2019-12-11T23:23:28.990344+0000) 2019-12-11T23:24:10.089 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:24:10.081+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e516 v516) 2019-12-11T23:24:10.537 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:24:10.372+0000 1694f700 -1 osd.0 517 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e513) 2019-12-11T23:24:11.094 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:24:11.022+0000 1694f700 -1 osd.7 517 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116611 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e509) 2019-12-11T23:24:41.379 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:41.212+0000 31786700 -1 osd.1 520 build_incremental_map_msg missing incremental map 520 2019-12-11T23:24:41.446 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:41.214+0000 31786700 -1 osd.1 520 build_incremental_map_msg missing incremental map 520 2019-12-11T23:24:41.498 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:41.421+0000 31786700 -1 osd.1 520 build_incremental_map_msg missing incremental map 520 2019-12-11T23:24:41.512 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:41.427+0000 35f8f700 -1 osd.1 520 build_incremental_map_msg missing incremental map 520 2019-12-11T23:24:41.617 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:41.584+0000 32788700 -1 osd.1 520 build_incremental_map_msg missing incremental map 520 2019-12-11T23:24:41.711 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:41.586+0000 32788700 -1 osd.1 520 build_incremental_map_msg missing incremental map 520 2019-12-11T23:24:42.217 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:24:42.181+0000 30f85700 -1 osd.1 520 build_incremental_map_msg missing incremental map 520 2019-12-11T23:25:18.684 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:18.680+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:19.723 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:19.719+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:20.751 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:20.740+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:21.742 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:21.738+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:22.781 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:22.777+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:23.797 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:23.793+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:24.774 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:24.770+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:25.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:25.804+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:26.861 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:26.858+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:27.847 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:27.844+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:28.827 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:28.824+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:29.879 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:29.876+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:30.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:30.926+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:31.939 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:31.937+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:32.974 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:32.971+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:34.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:34.001+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:34.987 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:34.983+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:35.952 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:35.949+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:36.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:36.930+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:37.914 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:37.910+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:38.905 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:38.901+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:39.659 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:53:26.591 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.667 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:53:26.599 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.676 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:53:26.608 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.679 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:53:26.611 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.694 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:53:26.598 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.725 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:53:26.628 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.827 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:53:26.759 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.849 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:53:26.781 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:39.913 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:39.909+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:39.916 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:53:26.819 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:25:40.912 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:40.910+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:41.915 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:41.911+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:42.938 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:42.934+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:43.391 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:43.386+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:43.964 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:43.960+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:44.227 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:44.222+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:44.373 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:44.369+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:44.585 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:44.582+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:44.920 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:44.913+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:44.920 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:44.916+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:45.188 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:45.184+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:45.366 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:45.360+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:45.548 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:45.544+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:45.595 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:45.590+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:45.958 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:45.952+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:45.958 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:45.955+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:46.028 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:46.024+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:46.211 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:46.206+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:46.333 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:46.328+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:46.523 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:46.520+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:46.585 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:46.580+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:46.923 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:46.914+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:46.924 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:46.917+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:47.074 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:47.071+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:47.196 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:47.193+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:47.352 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:47.348+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:47.555 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:47.551+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:47.597 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:47.593+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:47.962 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:47.957+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:47.963 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:47.959+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:48.071 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:48.066+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:48.205 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:48.199+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:48.346 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:48.339+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:48.571 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:48.568+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:48.600 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:48.593+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:48.927 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:48.923+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:48.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:48.927+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:49.098 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:49.093+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:49.239 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:49.235+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:49.363 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:49.358+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:49.607 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:49.603+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:49.652 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:49.647+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:49.892 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:49.888+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:49.892 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:49.889+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:50.126 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:50.122+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:50.263 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:50.259+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:50.396 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:50.390+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:50.611 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:50.606+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:50.640 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:50.636+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:50.855 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:50.851+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:50.857 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:50.854+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:51.122 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:51.118+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:51.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:51.230+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:51.375 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:51.369+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:51.641 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:51.638+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:51.657 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:51.652+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:51.906 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:51.900+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:51.906 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:51.902+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:52.083 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:52.078+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:52.215 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:52.212+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:52.364 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:52.359+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:52.657 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:52.652+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:52.661 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:52.657+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:52.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:52.875+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:52.882 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:52.878+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:53.092 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:53.085+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:53.196 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:53.187+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:53.331 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:53.325+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:53.629 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:53.624+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:53.676 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:53.669+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:53.857 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:53.853+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:53.860 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:53.857+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:54.072 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:54.066+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:54.209 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:54.204+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:54.358 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:54.352+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:54.606 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:54.602+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:54.640 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:54.634+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:54.876 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:54.870+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:54.876 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:54.872+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:55.086 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:55.081+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:55.259 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:55.253+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:55.344 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:55.339+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:55.583 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:55.580+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:55.674 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:55.668+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:55.903 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:55.897+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:55.904 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:55.900+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:56.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:56.043+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:56.298 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:56.293+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:56.381 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:56.376+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:56.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:56.598+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:56.678 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:56.673+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:56.910 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:56.904+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:56.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:56.907+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:57.029 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:57.024+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:57.321 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:57.316+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:57.359 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:57.353+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:57.614 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:57.611+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:57.728 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:57.706+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:57.920 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:57.914+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:57.920 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:57.917+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:58.027 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:58.020+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:58.321 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:58.315+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:58.322 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:58.317+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:58.576 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:58.571+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:58.734 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:58.729+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:58.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:58.932+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:58.940 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:58.936+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:59.029 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:59.023+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:25:59.307 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:25:59.304+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:25:59.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:25:59.329+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:25:59.578 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:25:59.575+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:25:59.732 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:25:59.726+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:25:59.952 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:59.946+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:25:59.952 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:25:59.948+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:25:59.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:25:59.989+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:00.343 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:00.337+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:00.352 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:00.349+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:00.536 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:00.532+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:00.718 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:00.712+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:00.948 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:00.944+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:00.969 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:00.963+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:00.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:00.966+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:01.370 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:01.365+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:01.400 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:01.396+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:01.572 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:01.568+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:01.701 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:01.695+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:01.944 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:01.939+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:01.987 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:01.981+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:01.988 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:01.983+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:02.360 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:02.357+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:02.404 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:02.399+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:02.619 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:02.616+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:02.726 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:02.720+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:02.908 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:02.902+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:03.017 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:03.010+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:03.024 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:03.014+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:03.344 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:03.341+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:03.375 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:03.370+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:03.603 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:03.599+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:03.693 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:03.688+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:03.866 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:03.863+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:04.011 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:04.005+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:04.012 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:04.008+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:04.338 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:04.333+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:04.395 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:04.390+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:04.630 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:04.627+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:04.717 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:04.712+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:04.887 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:04.882+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:05.057 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:05.050+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:05.057 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:05.053+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:05.384 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:05.381+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:05.444 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:05.437+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:05.628 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:05.624+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:05.687 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:05.682+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:05.879 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:05.875+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:06.054 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:06.047+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:06.061 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:06.053+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:06.432 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:06.429+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:06.496 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:06.491+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:06.648 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:06.645+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:06.683 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:06.678+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:06.866 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:06.862+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:07.088 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:07.084+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:07.093 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:07.087+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:07.475 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:07.471+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:07.536 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:07.530+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:07.643 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:07.640+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:07.717 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:07.711+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:07.854 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:07.849+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:08.078 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:08.074+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:08.079 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:08.075+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:08.470 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:08.465+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:08.556 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:08.550+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:08.661 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:08.657+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:08.713 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:08.705+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:08.889 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:08.884+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:09.049 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:09.042+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:09.050 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:09.046+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:09.452 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:09.449+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:09.610 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:09.604+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:09.689 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:09.683+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:09.710 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:09.706+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:09.859 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:09.855+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:10.051 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:10.045+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:10.052 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:10.048+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:10.449 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:10.445+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:10.640 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:10.634+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:10.742 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:10.735+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:10.746 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:10.743+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:10.827 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:10.822+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:11.029 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:11.023+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:11.029 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:11.025+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:11.424 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:11.421+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:11.680 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:11.676+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:11.742 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:11.737+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:11.756 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:11.751+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:11.813 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:11.808+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:12.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:12.054+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:12.062 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:12.057+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:12.381 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:12.378+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:12.635 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:12.629+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:12.717 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:12.713+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:12.765 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:12.760+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:12.836 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:12.833+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:13.114 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:13.108+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:13.115 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:13.111+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:13.350 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:13.347+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:13.632 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:13.628+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:13.721 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:13.716+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:13.762 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:13.757+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:13.808 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:13.804+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:14.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:14.113+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:14.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:14.114+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:14.329 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:14.325+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:14.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:14.679+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:14.743 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:14.740+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:14.778 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:14.774+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:14.829 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:14.826+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:15.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:15.143+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:15.150 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:15.146+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:15.334 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:15.331+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:15.477 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:26:15.473+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 45sec e521 v521) 2019-12-11T23:26:15.665 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:15.660+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:15.776 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:15.772+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:15.820 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:15.817+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:15.824 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:15.819+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:16.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:16.146+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:16.153 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:16.149+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:16.290 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:16.287+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:16.658 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:16.653+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:16.826 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:16.823+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:16.830 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:16.827+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:16.843 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:16.838+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:17.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:17.185+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:17.191 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:17.187+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:17.338 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:17.334+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:17.692 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:17.687+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:17.860 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:17.856+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:17.864 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:17.859+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:17.865 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:17.861+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:18.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:18.167+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:18.175 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:18.171+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:18.368 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:18.364+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:18.704 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:18.698+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:18.827 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:18.824+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:18.859 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:18.856+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:18.880 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:18.875+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:19.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:19.142+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:19.151 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:19.148+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:19.385 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:19.380+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:19.689 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:19.685+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:19.851 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:26:19.847+0000 1694f700 -1 osd.2 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.549798+0000 front 2019-12-11T23:25:00.672690+0000 (oldest deadline 2019-12-11T23:25:44.035986+0000) 2019-12-11T23:26:19.872 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:26:19.868+0000 1694f700 -1 osd.1 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:00.786441+0000 front 2019-12-11T23:25:00.614634+0000 (oldest deadline 2019-12-11T23:25:45.281165+0000) 2019-12-11T23:26:19.922 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:26:19.918+0000 1694f700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.464902+0000 front 2019-12-11T23:25:01.466355+0000 (oldest deadline 2019-12-11T23:25:44.873893+0000) 2019-12-11T23:26:20.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:20.138+0000 1694f700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:01.156489+0000 front 2019-12-11T23:25:00.700211+0000 (oldest deadline 2019-12-11T23:25:44.165256+0000) 2019-12-11T23:26:20.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:20.147+0000 1694f700 -1 osd.7 521 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:20.366 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:26:20.362+0000 1694f700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:24:59.125470+0000 front 2019-12-11T23:24:59.121061+0000 (oldest deadline 2019-12-11T23:25:43.759481+0000) 2019-12-11T23:26:20.535 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:26:20.530+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 45sec e521 v521) 2019-12-11T23:26:20.700 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:26:20.671+0000 1694f700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:25:03.041115+0000 front 2019-12-11T23:25:02.175263+0000 (oldest deadline 2019-12-11T23:25:43.010008+0000) 2019-12-11T23:26:21.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:21.474+0000 1694f700 -1 osd.7 522 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:22.499 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:22.494+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:23.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:23.614+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:24.608 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:24.603+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:25.615 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:25.613+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:26.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:26.639+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:27.686 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:27.684+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:28.715 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:28.712+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:29.740 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:29.738+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:30.745 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:30.743+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:30.843 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:54:17.775 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:30.861 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:54:17.793 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:30.899 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:54:17.831 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:30.901 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:54:17.833 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:30.952 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:54:17.855 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:30.959 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:54:17.891 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:30.964 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:54:17.868 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:30.999 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:54:17.930 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:31.082 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:54:17.985 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:26:31.748 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:31.746+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:32.734 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:32.731+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:33.757 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:33.754+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:34.762 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:34.759+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:35.781 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:35.777+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:36.760 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:36.756+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:37.803 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:37.799+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:38.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:38.848+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:39.843 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:39.839+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:40.803 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:40.799+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:41.805 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:41.802+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:42.844 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:42.840+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:43.829 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:43.824+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:44.857 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:44.852+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:45.869 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:45.865+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:46.897 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:46.894+0000 1694f700 -1 osd.7 523 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:47.885 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:47.881+0000 1694f700 -1 osd.7 524 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:48.912 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:48.909+0000 1694f700 -1 osd.7 524 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:50.319 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:50.314+0000 1694f700 -1 osd.7 525 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:51.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:51.361+0000 1694f700 -1 osd.7 526 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:52.450 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:52.388+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:52.906 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:52.881+0000 38f95700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:53.095 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:52.916+0000 36790700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:53.138 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:52.932+0000 36790700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:53.140 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:52.937+0000 36790700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:53.164 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:53.089+0000 38794700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:53.410 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:53.405+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:53.468 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:53.458+0000 33f8b700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:53.971 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:53.934+0000 38f95700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:54.755 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:54.462+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:55.186 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:26:55.171+0000 124b0700 -1 osd.0 527 build_incremental_map_msg missing incremental map 527 2019-12-11T23:26:55.468 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:55.465+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:56.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:56.423+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:57.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:57.386+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:58.362 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:58.358+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:26:59.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:26:59.360+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:00.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:00.369+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:01.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:01.382+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:02.349 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:02.345+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:03.334 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:03.330+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:04.354 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:04.350+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:05.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:05.397+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:06.418 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:06.411+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:07.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:07.368+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:08.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:08.351+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:09.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:09.347+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:10.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:10.353+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:11.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:11.406+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:12.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:12.382+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:13.439 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:13.432+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:14.469 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:14.466+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:15.424 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:15.420+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:16.443 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:16.440+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:17.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:17.436+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:18.396 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:18.391+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:18.476 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:18.471+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:18.477 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:18.473+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:19.193 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:19.189+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:19.219 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:19.214+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:19.436 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:19.430+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:19.489 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:19.484+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:19.490 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:19.486+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:19.971 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:19.968+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:20.198 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:20.194+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:20.253 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:20.246+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:20.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:20.398+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:20.461 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:20.456+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:20.461 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:20.458+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:21.002 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:20.998+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:21.239 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:21.234+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:21.243 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:21.238+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:21.365 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:21.361+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:21.425 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:21.420+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:21.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:21.422+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:22.055 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:22.050+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:22.209 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:22.197+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:22.252 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:22.246+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:22.397 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:22.389+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:22.400 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:22.390+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:22.401 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:22.392+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:23.106 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:23.101+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:23.232 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:23.228+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:23.240 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:23.238+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:23.370 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:23.367+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:23.375 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:23.370+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:23.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:23.422+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:24.072 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:24.067+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:24.238 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:24.234+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:24.257 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:24.253+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:24.285 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:24.282+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:24.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:24.389+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:24.402 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:24.390+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:24.430 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:24.426+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:25.075 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:25.071+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:25.222 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:25.218+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:25.274 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:25.270+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:25.290 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:25.287+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:25.392 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:25.384+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:25.395 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:25.393+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:25.406 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:25.402+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:26.110 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:26.105+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:26.228 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:26.224+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:26.297 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:26.293+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:26.325 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:26.322+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:26.363 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:26.359+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:26.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:26.436+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:26.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:26.438+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:27.129 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:27.126+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:27.252 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:27.249+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:27.281 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:27.278+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:27.351 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:27.347+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:27.363 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:27.359+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:27.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:27.426+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:27.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:27.427+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:28.140 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:28.137+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:28.261 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:28.255+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:28.261 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:28.258+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:28.285 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:28.278+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:28.320 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:28.317+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:28.367 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:28.364+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:28.405 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:28.401+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:28.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:28.406+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:29.139 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:29.136+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:29.267 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:29.263+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:29.269 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:29.267+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:29.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:29.292+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:29.308 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:29.303+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:29.420 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:29.416+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:29.421 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:29.418+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:29.422 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:29.418+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:30.123 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:30.120+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:30.260 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:30.256+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:30.263 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:30.260+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:30.265 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:30.261+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:30.335 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:30.332+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:30.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:30.387+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:30.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:30.389+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:30.462 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:30.458+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:31.149 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:31.145+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:31.285 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:31.281+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:31.315 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:31.311+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:31.408 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:31.365+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:31.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:31.393+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:31.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:31.397+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:31.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:31.312+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:31.428 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:31.424+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:32.119 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:32.116+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:32.251 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:32.248+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:32.302 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:32.299+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:32.304 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:32.302+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:32.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:32.357+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:32.397 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:32.392+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:32.398 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:32.394+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:32.442 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:32.437+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:33.095 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:33.091+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:33.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:33.294+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:33.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:33.312+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:33.319 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:33.315+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:33.388 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:33.381+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:33.438 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:33.434+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:33.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:33.437+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:33.488 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:33.484+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:34.124 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:34.117+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:34.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:34.291+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:34.344 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:34.340+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:34.346 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:34.344+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:34.354 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:34.348+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:34.484 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:34.480+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:34.487 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:34.484+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:34.517 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:34.512+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:35.127 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:35.123+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:35.303 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:35.299+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:35.315 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:35.311+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:35.315 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:35.312+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:35.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:35.323+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:35.444 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:35.440+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:35.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:35.443+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:35.504 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:35.496+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:36.131 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:36.127+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:36.265 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:36.261+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:36.294 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:36.290+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:36.294 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:36.291+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:36.354 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:36.349+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:36.465 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:36.460+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:36.466 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:36.462+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:36.470 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:36.466+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:37.150 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:37.146+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:37.236 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:37.232+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:37.294 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:37.289+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:37.294 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:37.291+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:37.339 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:37.333+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:37.444 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:37.439+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:37.510 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:37.505+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:37.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:37.507+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:38.106 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:38.102+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:38.243 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:38.240+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:38.284 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:38.281+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:38.285 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:38.282+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:38.354 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:38.349+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:38.423 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:38.419+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:38.467 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:38.462+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:38.468 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:38.464+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:39.137 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:39.133+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:39.253 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:39.250+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:39.327 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:39.322+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:39.328 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:39.324+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:39.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:39.333+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:39.425 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:39.421+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:39.498 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:39.491+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:39.498 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:39.493+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:40.099 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:40.096+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:40.285 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:40.281+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:40.298 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:40.294+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:40.333 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:40.329+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:40.334 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:40.330+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:40.415 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:40.409+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:40.555 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:40.550+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:40.556 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:40.552+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:41.073 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:41.071+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:41.305 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:41.300+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:41.321 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:41.317+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:41.328 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:41.325+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:41.330 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:41.328+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:41.422 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:41.417+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:41.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:41.579+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:41.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:41.581+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:42.059 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:42.055+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:42.293 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:42.289+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:42.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:42.314+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:42.377 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:42.372+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:42.378 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:42.374+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:42.463 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:42.457+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:42.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:42.556+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:42.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:42.557+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:43.059 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:43.055+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:43.284 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:43.280+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:43.330 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:43.326+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:43.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:43.327+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:43.337 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:43.333+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:43.483 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:43.478+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:43.535 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:43.531+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:43.539 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:43.536+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:44.051 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:44.047+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:44.275 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:44.270+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:44.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:44.293+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:44.376 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:44.371+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:44.376 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:44.373+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:44.493 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:44.488+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:44.498 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:44.494+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:44.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:44.497+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:45.102 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:45.098+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:45.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:45.324+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:45.335 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:45.332+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:45.346 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:45.343+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:45.348 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:45.346+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:45.462 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:45.457+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:45.467 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:45.463+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:45.543 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:45.538+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:46.153 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:46.150+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:46.289 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:46.284+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:46.303 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:46.298+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:46.356 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:46.352+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:46.357 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:46.353+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:46.430 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:46.426+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:46.433 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:46.430+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:46.536 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:46.532+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:47.164 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:47.161+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:47.267 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:47.262+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:47.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:47.330+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:47.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:47.332+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:47.343 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:47.339+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:47.439 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:47.434+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:47.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:47.436+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:47.526 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:47.512+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:48.189 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:48.185+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:48.312 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:48.307+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:48.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:48.328+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:48.332 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:48.329+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:48.362 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:48.358+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:48.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:48.403+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:48.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:48.408+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:48.554 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:48.550+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:49.171 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:49.167+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:49.289 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:49.286+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:49.291 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:49.289+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:49.360 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:49.352+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:49.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:49.360+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:49.366 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:49.362+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:49.367 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:49.361+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:49.553 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:49.548+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:50.172 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:50.169+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:50.329 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:50.326+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:50.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:50.332+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:50.336 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:50.333+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:50.353 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:50.341+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:50.377 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:50.373+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:50.380 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:50.377+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:50.598 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:50.594+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:50.988 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:27:50.985+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e527 v527) 2019-12-11T23:27:51.157 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:51.154+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:51.298 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:51.295+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:51.301 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:51.298+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:51.329 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:51.326+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:51.376 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:51.371+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:51.398 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:51.393+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:51.398 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:51.395+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:51.576 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:51.568+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:52.155 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:52.152+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:52.292 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:52.289+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:52.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:52.314+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:52.318 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:52.315+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:52.374 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:52.369+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:52.407 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:52.402+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:52.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:52.404+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:52.580 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:52.574+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:53.185 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:53.182+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:53.295 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:53.291+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:53.296 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:53.293+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:53.324 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:53.321+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:53.366 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:53.361+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:53.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:53.407+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:53.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:53.411+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:53.578 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:53.574+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:54.239 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:54.234+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:54.254 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:54.250+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:54.254 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:54.251+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:54.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:54.348+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:54.385 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:54.376+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:54.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:54.405+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:54.410 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:54.406+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:54.580 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:54.575+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:55.256 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:55.251+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:55.283 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:55.280+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:55.285 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:55.283+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:55.401 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:55.395+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:55.423 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:55.418+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:55.424 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:55.420+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:55.429 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:55.425+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:55.607 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:55.603+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:56.002 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:27:55.998+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e527 v527) 2019-12-11T23:27:56.299 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:56.296+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:56.321 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:56.318+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:56.322 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:56.320+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:56.399 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:56.388+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:56.450 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:56.445+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:56.450 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:56.446+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:56.452 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:56.449+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:56.621 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:56.616+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:57.295 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:57.292+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:57.348 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:57.345+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:57.349 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:57.346+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:57.442 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:57.437+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:57.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:57.467+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:57.474 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:57.471+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:57.495 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:57.492+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:57.608 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:57.603+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:58.333 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:58.329+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:58.333 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:58.330+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:58.349 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:58.345+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:58.425 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:58.416+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:58.512 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:58.507+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:58.513 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:58.509+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:27:58.522 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:58.519+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:58.567 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:58.562+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:59.335 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:27:59.332+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:27:59.349 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:59.345+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:27:59.349 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:27:59.346+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:27:59.391 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:27:59.386+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:27:59.531 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:27:59.526+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:27:59.538 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:27:59.534+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:27:59.550 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:59.545+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:27:59.550 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:27:59.547+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:00.320 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:28:00.317+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:28:00.376 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:00.370+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:28:00.377 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:00.371+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:28:00.409 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:28:00.404+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:28:00.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:28:00.490+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:28:00.565 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:28:00.560+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:28:00.587 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:00.582+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:28:00.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:00.584+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:01.023 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:28:01.019+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e527 v527) 2019-12-11T23:28:01.369 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:28:01.366+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:28:01.413 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:28:01.408+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:28:01.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:01.418+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:28:01.422 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:01.419+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:28:01.513 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:28:01.509+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:28:01.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:01.543+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:28:01.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:01.544+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:01.567 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:28:01.563+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:28:02.385 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:02.381+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:28:02.386 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:02.383+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:28:02.397 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:28:02.392+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:28:02.422 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:28:02.419+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:28:02.526 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:28:02.521+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:28:02.536 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:02.532+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:28:02.537 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:02.533+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:02.567 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:28:02.563+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:28:03.384 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:03.378+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:28:03.384 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:03.381+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:28:03.421 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:28:03.416+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:28:03.447 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:28:03.444+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:28:03.502 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:28:03.494+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:28:03.555 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:28:03.551+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:28:03.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:03.577+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:28:03.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:03.579+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:04.417 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:04.413+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:28:04.420 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:04.417+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:28:04.430 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:28:04.425+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:28:04.452 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:28:04.447+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:28:04.506 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:28:04.500+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:28:04.565 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:28:04.561+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:28:04.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:04.569+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:28:04.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:04.570+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:05.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:05.384+0000 1694f700 -1 osd.0 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:26:48.048442+0000 (oldest deadline 2019-12-11T23:27:28.048442+0000) 2019-12-11T23:28:05.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:05.389+0000 1694f700 -1 osd.0 527 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:28:05.424 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:28:05.419+0000 1694f700 -1 osd.1 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.200567+0000 front 2019-12-11T23:26:36.317007+0000 (oldest deadline 2019-12-11T23:27:19.085005+0000) 2019-12-11T23:28:05.473 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:28:05.468+0000 1694f700 -1 osd.6 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.073409+0000 front 2019-12-11T23:26:36.071504+0000 (oldest deadline 2019-12-11T23:27:18.311415+0000) 2019-12-11T23:28:05.532 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:28:05.523+0000 1694f700 -1 osd.4 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:38.066588+0000 front 2019-12-11T23:26:38.044698+0000 (oldest deadline 2019-12-11T23:27:19.097347+0000) 2019-12-11T23:28:05.568 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:28:05.563+0000 1694f700 -1 osd.2 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:36.599700+0000 front 2019-12-11T23:26:36.548547+0000 (oldest deadline 2019-12-11T23:27:18.784272+0000) 2019-12-11T23:28:05.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:05.568+0000 1694f700 -1 osd.7 527 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:26:37.550656+0000 front 2019-12-11T23:26:37.583955+0000 (oldest deadline 2019-12-11T23:27:18.373000+0000) 2019-12-11T23:28:05.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:05.570+0000 1694f700 -1 osd.7 527 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:06.089 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:28:06.080+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e527 v527) 2019-12-11T23:28:06.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:28:06.414+0000 1694f700 -1 osd.0 528 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e526) 2019-12-11T23:28:06.726 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:28:06.589+0000 1694f700 -1 osd.7 528 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:116985 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs out=1025b,stat out=16b,read 0~141] snapc 0=[] ondisk+read+known_if_redirected e521) 2019-12-11T23:28:58.976 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:28:58.970+0000 31786700 -1 osd.1 532 build_incremental_map_msg missing incremental map 520 2019-12-11T23:29:00.806 INFO:teuthology.orchestra.run.smithi079.stderr:testIfUnmodifiedSince (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-11T23:29:21.493 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:57:08.416 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.493 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:57:08.419 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.535 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:57:08.438 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.550 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:57:08.453 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.561 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:57:08.489 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.561 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:57:08.492 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.702 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:57:08.634 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.774 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:57:08.706 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:21.836 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:57:08.740 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:29:33.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:33.573+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:34.557 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:34.553+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:35.460 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:35.455+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:35.590 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:35.583+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:35.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:35.586+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:35.691 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:35.687+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:36.494 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:36.490+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:36.616 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:36.610+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:36.616 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:36.613+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:36.746 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:36.741+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:37.490 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:37.486+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:37.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:37.595+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:37.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:37.597+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:37.762 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:37.757+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:38.380 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:38.375+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:38.512 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:38.505+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:38.621 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:38.615+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:38.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:38.618+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:38.715 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:38.709+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:38.729 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:38.723+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:39.395 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:39.390+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:39.536 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:39.533+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:39.578 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:39.574+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:39.579 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:39.576+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:39.674 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:39.669+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:39.765 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:39.762+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:40.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:40.364+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:40.490 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:40.487+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:40.544 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:40.539+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:40.545 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:40.542+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:40.703 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:40.697+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:40.762 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:40.759+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:41.343 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:41.339+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:41.497 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:41.494+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:41.546 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:41.541+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:41.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:41.544+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:41.738 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:41.734+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:41.795 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:41.792+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:42.322 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:42.317+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:42.502 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:42.499+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:42.578 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:42.573+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:42.580 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:42.578+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:42.749 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:42.745+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:42.781 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:42.775+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:43.337 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:43.332+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:43.498 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:43.494+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:43.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:43.598+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:43.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:43.601+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:43.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:43.777+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:43.806 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:43.800+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:44.320 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:44.315+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:44.505 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:44.501+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:44.647 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:44.641+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:44.653 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:44.644+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:44.786 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:44.780+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:44.792 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:44.787+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:45.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:45.368+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:45.507 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:45.504+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:45.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:45.612+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:45.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:45.614+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:45.766 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:45.761+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:45.800 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:45.797+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:46.384 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:46.379+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:46.508 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:46.504+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:46.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:46.601+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:46.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:46.604+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:46.795 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:46.789+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:46.803 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:46.800+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:47.352 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:47.348+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:47.526 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:47.522+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:47.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:47.614+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:47.620 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:47.616+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:47.755 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:47.751+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:47.796 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:47.792+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:47.804 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:47.801+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:48.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:48.356+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:48.545 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:48.542+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:48.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:48.612+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:48.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:48.613+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:48.740 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:48.734+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:48.767 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:48.763+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:48.793 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:48.790+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:49.415 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:49.409+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:49.505 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:49.502+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:49.609 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:49.605+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:49.613 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:49.609+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:49.703 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:49.698+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:49.814 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:49.809+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:49.842 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:49.839+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:50.389 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:50.384+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:50.542 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:50.537+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:50.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:50.594+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:50.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:50.597+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:50.710 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:50.706+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:50.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:50.802+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:50.824 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:50.820+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:51.363 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:51.358+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:51.547 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:51.544+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:51.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:51.613+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:51.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:51.616+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:51.730 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:51.725+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:51.774 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:51.771+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:51.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:51.796+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:52.353 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:52.348+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:52.531 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:52.528+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:52.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:52.619+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:52.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:52.621+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:52.727 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:52.722+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:52.730 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:52.728+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:52.771 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:52.768+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:53.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:53.398+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:53.498 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:53.495+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:53.622 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:53.617+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:53.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:53.619+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:53.752 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:53.749+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:53.759 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:53.753+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:53.767 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:53.764+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:54.454 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:54.449+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:54.531 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:54.525+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:54.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:54.593+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:54.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:54.598+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:54.750 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:54.748+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:54.795 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:54.788+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:54.814 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:54.811+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:55.447 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:55.443+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:55.516 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:55.513+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:55.605 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:55.600+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:55.608 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:55.605+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:55.715 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:55.711+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:55.804 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:55.799+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:55.825 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:55.822+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:56.422 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:56.417+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:56.500 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:56.498+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:56.594 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:56.589+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:56.595 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:56.592+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:56.735 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:56.732+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:56.768 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:56.764+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:56.851 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:56.848+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:57.378 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:57.372+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:57.469 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:57.462+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:57.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:57.568+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:57.575 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:57.573+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:57.747 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:57.743+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:57.776 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:57.771+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:57.818 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:57.814+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:58.404 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:58.400+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:58.510 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:58.506+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:58.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:58.577+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:58.585 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:58.582+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:58.780 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:58.777+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:58.793 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:58.789+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:29:58.810 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:58.805+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:59.457 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:29:59.453+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:29:59.521 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:29:59.518+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:29:59.558 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:59.551+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:29:59.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:29:59.553+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:29:59.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:29:59.791+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:29:59.802 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:29:59.797+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:29:59.828 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:29:59.823+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:00.490 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:00.484+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:00.498 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:00.491+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:00.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:00.599+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:00.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:00.604+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:00.821 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:00.816+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:00.836 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:00.831+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:00.877 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:00.874+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:01.514 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:01.510+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:01.544 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:01.541+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:01.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:01.567+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:01.573 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:01.569+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:01.810 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:01.805+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:01.833 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:01.829+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:01.918 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:01.914+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:02.519 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:02.515+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:02.569 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:02.563+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:02.570 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:02.566+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:02.590 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:02.586+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:02.792 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:02.788+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:02.797 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:02.792+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:02.952 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:02.946+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:03.540 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:03.534+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:03.545 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:03.542+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:03.608 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:03.602+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:03.608 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:03.605+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:03.754 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:03.749+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:03.784 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:03.781+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:03.955 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:03.950+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:04.505 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:04.499+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:04.559 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:04.556+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:04.594 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:04.590+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:04.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:04.593+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:04.760 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:04.756+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:04.801 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:04.796+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:04.990 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:04.986+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:05.522 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:05.517+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:05.597 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:05.594+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:05.599 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:05.595+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:05.603 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:05.600+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:05.775 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:05.772+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:05.794 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:05.789+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:05.967 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:05.962+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:06.446 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:30:06.441+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 41sec e532 v532) 2019-12-11T23:30:06.500 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:06.495+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:06.621 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:06.618+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:06.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:06.627+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:06.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:06.630+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:06.791 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:06.789+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:06.801 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:06.795+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:06.978 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:06.974+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:07.467 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:07.462+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:07.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:07.582+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:07.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:07.585+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:07.648 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:07.645+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:07.758 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:07.753+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:07.788 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:07.785+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:08.023 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:08.020+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:08.515 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:08.506+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:08.622 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:08.617+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:08.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:08.627+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:08.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:08.630+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:08.775 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:08.770+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:08.789 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:08.786+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:09.001 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:08.998+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:09.527 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:09.522+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:09.603 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:09.600+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:09.662 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:09.657+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:09.666 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:09.663+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:09.788 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:09.784+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:09.815 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:09.809+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:09.992 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:09.988+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:10.555 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:10.550+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:10.627 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:30:10.623+0000 1694f700 -1 osd.1 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:54.307808+0000 front 2019-12-11T23:28:54.406228+0000 (oldest deadline 2019-12-11T23:29:35.375817+0000) 2019-12-11T23:30:10.693 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:10.688+0000 1694f700 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.631642+0000 front 2019-12-11T23:28:52.672976+0000 (oldest deadline 2019-12-11T23:29:34.912699+0000) 2019-12-11T23:30:10.696 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:10.693+0000 1694f700 -1 osd.7 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:10.773 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:30:10.769+0000 1694f700 -1 osd.2 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.966507+0000 front 2019-12-11T23:28:53.968242+0000 (oldest deadline 2019-12-11T23:29:47.602734+0000) 2019-12-11T23:30:10.860 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:30:10.855+0000 1694f700 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:52.792185+0000 front 2019-12-11T23:28:52.572798+0000 (oldest deadline 2019-12-11T23:29:38.309345+0000) 2019-12-11T23:30:11.025 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:30:11.022+0000 1694f700 -1 osd.3 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.691725+0000 front 2019-12-11T23:28:53.917015+0000 (oldest deadline 2019-12-11T23:29:35.380799+0000) 2019-12-11T23:30:11.499 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:30:11.496+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 41sec e532 v532) 2019-12-11T23:30:11.551 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:30:11.546+0000 1694f700 -1 osd.6 532 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:28:53.337083+0000 front 2019-12-11T23:28:53.404233+0000 (oldest deadline 2019-12-11T23:29:37.413781+0000) 2019-12-11T23:30:12.030 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:11.704+0000 1694f700 -1 osd.7 533 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:12.863 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:12.855+0000 1694f700 -1 osd.7 533 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:13.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:13.894+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:14.944 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:14.917+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:15.951 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:15.947+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:16.953 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:16.950+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:17.912 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:17.907+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:18.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:18.931+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:19.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:19.942+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:20.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:20.931+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:21.960 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:21.957+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:22.942 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:22.938+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:23.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:23.907+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:24.949 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:24.947+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:25.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:25.922+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:26.938 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:26.935+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:27.908 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:27.904+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:28.877 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:28.873+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:29.840 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:29.837+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:30.802 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:30.798+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:31.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:31.853+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:32.856 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:32.852+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:33.826 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:33.822+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:34.843 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:34.839+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:35.846 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:35.843+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:36.841 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:36.837+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:37.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:37.804+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:38.795 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:38.792+0000 1694f700 -1 osd.7 534 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:39.828 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:39.825+0000 1694f700 -1 osd.7 535 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:40.791 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:40.787+0000 1694f700 -1 osd.7 535 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:41.751 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:41.748+0000 1694f700 -1 osd.7 535 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:43.197 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:43.060+0000 1694f700 -1 osd.7 536 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:44.067 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:44.062+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:45.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:45.069+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:46.103 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:46.101+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:47.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:47.235+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:48.325 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:48.315+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:49.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:49.338+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:50.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:50.307+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:51.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:51.313+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:52.331 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:52.327+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:53.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:53.316+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:54.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:54.366+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:55.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:55.406+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:56.374 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:56.365+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:57.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:57.404+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:58.401 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:58.398+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:30:59.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:30:59.436+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:00.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:00.405+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:01.415 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:01.412+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:02.402 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:02.399+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:03.407 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:03.404+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:04.380 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:04.377+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:05.416 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:05.413+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:06.371 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:06.367+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:07.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:07.329+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:08.333 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:08.329+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:09.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:09.320+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:09.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:09.970+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:10.285 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:10.277+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:11.026 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:11.022+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:11.066 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:11.059+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:11.308 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:11.304+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:11.335 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:11.332+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:12.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:12.013+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:12.048 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:12.045+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:12.308 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:12.304+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:12.335 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:12.331+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:12.344 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:12.340+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:12.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:12.971+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:13.088 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:13.084+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:13.309 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:13.304+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:13.332 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:13.327+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:13.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:13.340+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:13.346 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:13.342+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:14.014 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:14.011+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:14.051 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:14.045+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:14.332 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:14.327+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:14.348 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:14.343+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:14.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:14.349+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:14.355 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:14.353+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:14.571 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:14.569+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:15.009 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:15.005+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:15.033 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:15.029+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:15.312 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:15.308+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:15.368 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:15.361+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:15.368 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:15.362+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:15.379 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:15.372+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:15.578 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:15.574+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:15.990 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:15.988+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:16.045 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:16.033+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:16.315 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:16.310+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:16.353 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:16.340+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:16.368 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:16.359+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:16.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:16.360+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:16.561 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:16.559+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:16.968 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:16.964+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:17.059 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:17.055+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:17.287 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:17.283+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:17.359 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:17.356+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:17.363 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:17.361+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:17.393 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:17.389+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:17.594 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:17.591+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:17.947 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:17.943+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:18.035 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:18.030+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:18.293 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:18.288+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:18.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:18.375+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:18.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:18.377+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:18.406 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:18.403+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:18.597 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:18.589+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:18.921 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:18.916+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:18.990 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:18.986+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:19.302 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:19.298+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:19.347 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:19.343+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:19.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:19.345+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:19.398 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:19.394+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:19.608 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:19.593+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:19.921 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:19.917+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:19.988 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:19.985+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:20.271 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:20.266+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:20.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:20.319+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:20.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:20.321+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:20.407 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:20.404+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:20.561 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:20.557+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:20.564 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:20.561+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:20.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:20.931+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:21.019 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:21.015+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:21.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:21.316+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:21.359 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:21.356+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:21.362 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:21.360+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:21.390 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:21.387+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:21.602 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:21.599+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:21.602 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:21.600+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:21.947 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:21.944+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:21.978 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:21.973+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:22.300 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:22.297+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:22.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:22.357+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:22.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:22.359+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:22.409 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:22.403+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:22.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:22.569+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:22.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:22.571+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:22.947 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:22.945+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:23.017 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:23.015+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:23.303 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:23.300+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:23.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:23.361+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:23.367 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:23.365+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:23.460 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:23.457+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:23.568 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:23.564+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:23.569 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:23.567+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:23.969 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:23.963+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:24.023 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:24.016+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:24.261 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:24.256+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:24.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:24.335+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:24.341 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:24.336+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:24.471 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:24.466+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:24.592 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:24.573+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:24.592 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:24.576+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:24.999 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:24.995+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:25.024 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:25.020+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:25.312 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:25.308+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:25.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:25.339+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:25.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:25.341+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:25.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:25.446+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:25.543 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:25.540+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:25.544 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:25.541+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:26.022 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:26.018+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:26.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:26.045+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:26.306 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:26.301+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:26.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:26.383+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:26.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:26.385+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:26.413 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:26.408+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:26.533 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:26.530+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:26.535 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:26.532+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:27.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:27.016+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:27.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:27.083+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:27.291 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:27.287+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:27.361 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:27.357+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:27.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:27.361+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:27.377 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:27.373+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:27.575 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:27.572+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:27.576 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:27.574+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:28.012 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:28.010+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:28.067 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:28.064+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:28.336 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:28.332+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:28.352 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:28.347+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:28.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:28.404+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:28.411 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:28.408+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:28.549 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:28.544+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:28.549 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:28.545+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:28.973 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:28.968+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:29.117 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:29.111+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:29.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:29.314+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:29.358 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:29.354+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:29.458 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:29.450+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:29.459 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:29.452+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:29.566 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:29.550+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:29.566 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:29.551+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:29.981 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:29.976+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:30.136 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:30.134+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:30.345 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:30.341+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:30.373 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:30.368+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:30.476 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:30.466+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:30.476 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:30.468+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:30.563 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:30.561+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:30.565 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:30.563+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:30.944 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:30.941+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:31.186 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:31.182+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:31.323 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:31.319+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:31.378 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:31.374+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:31.431 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:31.426+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:31.432 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:31.427+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:31.608 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:31.604+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:31.608 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:31.606+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:31.961 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:31.959+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:32.173 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:32.168+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:32.303 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:32.299+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:32.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:32.422+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:32.476 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:32.472+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:32.477 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:32.474+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:32.576 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:32.574+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:32.577 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:32.575+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:32.994 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:32.991+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:33.193 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:33.190+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:33.351 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:33.343+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:33.420 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:33.415+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:33.494 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:33.489+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:33.495 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:33.491+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:33.551 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:33.547+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:33.552 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:33.548+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:33.961 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:33.959+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:34.182 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:34.179+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:34.321 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:34.317+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:34.420 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:34.406+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:34.514 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:34.510+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:34.514 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:34.511+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:34.589 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:34.585+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:34.589 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:34.586+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:34.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:34.939+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:35.237 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:35.233+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:35.299 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:35.287+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:35.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:35.367+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:35.519 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:35.514+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:35.519 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:35.515+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:35.621 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:35.616+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:35.622 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:35.618+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:35.982 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:35.978+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:36.261 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:36.258+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:36.322 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:36.318+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:36.343 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:36.338+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:36.556 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:36.553+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:36.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:36.556+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:36.655 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:36.650+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:36.655 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:36.653+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:36.958 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:36.955+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:37.233 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:37.229+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:37.303 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:37.298+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:37.341 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:37.336+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:37.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:37.569+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:37.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:37.571+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:37.660 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:37.657+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:37.661 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:37.658+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:38.002 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:38.000+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:38.236 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:38.233+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:38.279 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:38.275+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:38.296 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:38.291+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:38.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:38.587+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:38.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:38.589+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:38.642 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:38.639+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:38.643 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:38.640+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:39.036 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:39.032+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:39.253 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:39.248+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:39.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:39.260+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:39.287 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:39.284+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:39.611 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:39.606+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:39.611 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:39.608+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:39.615 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:39.612+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:39.617 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:39.615+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:40.053 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:40.049+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:40.233 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:40.229+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:40.252 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:40.248+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:40.316 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:40.312+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:40.622 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:40.617+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:40.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:40.619+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:40.653 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:40.650+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:40.654 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:40.651+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:41.073 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:41.070+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:41.272 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:41.268+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:41.275 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:41.271+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:41.296 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:41.292+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:41.587 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:41.583+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:41.590 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:41.587+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:41.646 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:41.643+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:41.647 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:41.644+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:41.941 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:59:28.861 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:41.946 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:59:28.875 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:41.953 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:59:28.885 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:41.960 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:59:28.890 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:41.999 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:59:28.902 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:42.000 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:59:28.904 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:42.066 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:42.063+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:42.098 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:06:59:29.030 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:42.125 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:06:59:29.058 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:42.167 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:06:59:29.070 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:31:42.239 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:42.233+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:42.260 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:42.258+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:42.303 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:42.299+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:42.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:42.596+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:42.602 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:42.598+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:42.623 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:42.619+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:42.624 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:42.621+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:43.032 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:43.029+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:43.215 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:43.207+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:43.280 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:43.277+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:43.295 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:43.291+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:43.584 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:43.582+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:43.585 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:43.583+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:43.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:43.594+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:43.611 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:43.596+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:44.081 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:44.076+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:44.259 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:44.255+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:44.276 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:44.271+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:44.298 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:44.294+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:44.597 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:44.593+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:44.598 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:44.594+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:44.614 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:44.610+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:44.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:44.614+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:45.116 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:45.112+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:45.255 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:45.251+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:45.323 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:45.319+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:45.328 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:45.323+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:45.626 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:45.622+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:45.627 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:45.623+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:45.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:45.636+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:45.642 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:45.638+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:46.082 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:46.079+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:46.282 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:46.277+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:46.290 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:46.286+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:46.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:46.315+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:46.645 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:46.637+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:46.648 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:46.643+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:46.657 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:46.653+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:46.657 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:46.654+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:46.899 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:31:46.896+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 48sec e537 v537) 2019-12-11T23:31:47.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:47.057+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:47.276 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:47.272+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:47.300 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:47.296+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:47.339 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:47.331+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:47.656 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:47.651+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:47.656 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:47.653+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:47.702 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:47.699+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:47.704 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:47.701+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:48.023 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:48.019+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:48.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:48.259+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:48.277 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:48.273+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:48.334 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:48.329+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:48.629 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:48.619+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:48.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:48.623+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:48.722 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:48.718+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:48.722 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:48.719+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:48.994 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:48.990+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:49.306 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:49.299+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:49.309 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:49.306+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:49.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:49.313+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:49.634 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:49.629+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:49.635 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:49.631+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:49.684 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:49.680+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:49.685 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:49.682+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:50.033 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:50.029+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:50.280 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:50.275+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:50.330 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:50.326+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:50.341 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:50.338+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:50.687 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:50.682+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:50.687 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:50.684+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:50.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:50.693+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:50.700 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:50.695+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:51.084 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:51.080+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:51.285 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:51.281+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:51.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:51.310+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:51.351 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:51.349+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:51.703 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:51.700+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:51.703 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:51.701+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:51.704 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:51.698+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:51.704 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:51.700+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:51.918 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:31:51.916+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 48sec e537 v537) 2019-12-11T23:31:52.103 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:52.100+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:52.285 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:52.281+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:52.291 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:52.287+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:52.342 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:52.339+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:52.672 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:52.667+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:52.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:52.669+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:52.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:52.731+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:52.736 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:52.733+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:53.060 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:53.057+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:53.301 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:53.296+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:53.310 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:53.306+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:53.346 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:53.342+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:53.683 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:53.679+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:53.686 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:53.683+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:53.771 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:53.764+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:53.771 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:53.766+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:54.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:54.063+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:54.328 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:54.324+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:54.344 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:54.339+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:54.346 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:54.342+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:54.676 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:54.672+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:54.677 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:54.673+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:54.728 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:54.725+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:54.728 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:54.726+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:55.052 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:55.049+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:55.325 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:55.322+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:55.331 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:55.327+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:55.334 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:55.330+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:55.676 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:55.672+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:55.679 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:55.676+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:55.779 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:55.776+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:55.780 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:55.777+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:56.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:56.016+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:31:56.362 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:31:56.358+0000 1694f700 -1 osd.1 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:23.395939+0000 front 2019-12-11T23:30:23.394844+0000 (oldest deadline 2019-12-11T23:31:09.268099+0000) 2019-12-11T23:31:56.368 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:31:56.363+0000 1694f700 -1 osd.4 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.898516+0000 front 2019-12-11T23:30:28.892230+0000 (oldest deadline 2019-12-11T23:31:11.492527+0000) 2019-12-11T23:31:56.384 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:31:56.380+0000 1694f700 -1 osd.6 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.356621+0000 front 2019-12-11T23:30:27.395633+0000 (oldest deadline 2019-12-11T23:31:10.802517+0000) 2019-12-11T23:31:56.714 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:56.709+0000 1694f700 -1 osd.7 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:27.434214+0000 front 2019-12-11T23:30:27.459305+0000 (oldest deadline 2019-12-11T23:31:12.508926+0000) 2019-12-11T23:31:56.714 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:31:56.711+0000 1694f700 -1 osd.7 537 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117420 2.2fs0 2:f65724b3:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ce%a9%e0%a4%80%ed%88%8e%e1%a0%83%ed%88%8e%e0%a4%83%e0%a4%83%ef%84%91%ed%88%8f%e9%90%8b%e1%a0%81%e0%a4%81%e2%98%83%e8%94%89%e8%94%8a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e532) 2019-12-11T23:31:56.783 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:56.780+0000 1694f700 -1 osd.0 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:30:39.660308+0000 (oldest deadline 2019-12-11T23:31:19.660308+0000) 2019-12-11T23:31:56.785 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:31:56.783+0000 1694f700 -1 osd.0 537 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e536) 2019-12-11T23:31:56.966 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:31:56.959+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 48sec e537 v537) 2019-12-11T23:31:57.054 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:31:57.050+0000 1694f700 -1 osd.2 537 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:30:28.890420+0000 front 2019-12-11T23:30:28.590862+0000 (oldest deadline 2019-12-11T23:31:10.805424+0000) 2019-12-11T23:32:25.602 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:32:25.585+0000 3578e700 -1 osd.3 543 build_incremental_map_msg missing incremental map 542 2019-12-11T23:33:01.868 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:00:48.800 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:01.874 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:00:48.806 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:01.882 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:00:48.815 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:01.887 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:00:48.818 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:01.900 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:00:48.804 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:01.912 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:00:48.816 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:02.039 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:00:48.972 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:02.062 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:00:48.984 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:02.062 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:00:48.953 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:33:22.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:22.936+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:23.746 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:23.737+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:23.903 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:23.899+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:24.721 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:24.716+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:24.929 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:24.922+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:24.991 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:24.988+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:25.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:25.246+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:25.510 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:25.507+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:25.724 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:25.719+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:25.965 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:25.953+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:25.965 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:25.956+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:25.997 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:25.994+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:26.243 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:26.233+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:26.517 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:26.515+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:26.756 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:26.751+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:27.002 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:26.998+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:27.005 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:27.002+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:27.006 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:27.003+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:27.142 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:27.139+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:27.208 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:27.203+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:27.503 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:27.501+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:27.731 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:27.728+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:27.979 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:27.974+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:27.980 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:27.977+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:28.053 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:28.050+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:28.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:28.115+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:28.231 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:28.226+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:28.540 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:28.536+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:28.744 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:28.740+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:28.975 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:28.970+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:28.976 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:28.973+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:29.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:29.012+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:29.100 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:29.097+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:29.279 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:29.272+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:29.580 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:29.576+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:29.744 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:29.740+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:29.929 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:29.925+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:29.932 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:29.930+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:30.067 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:30.064+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:30.084 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:30.080+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:30.321 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:30.316+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:30.567 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:30.564+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:30.748 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:30.742+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:30.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:30.967+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:30.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:30.970+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:31.117 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:31.113+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:31.122 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:31.118+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:31.322 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:31.318+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:31.595 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:31.591+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:31.782 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:31.777+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:31.967 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:31.961+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:31.967 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:31.964+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:32.095 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:32.091+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:32.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:32.126+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:32.367 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:32.362+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:32.604 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:32.600+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:32.747 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:32.743+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:32.949 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:32.945+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:32.950 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:32.947+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:33.089 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:33.085+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:33.127 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:33.122+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:33.384 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:33.379+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:33.587 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:33.583+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:33.740 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:33.733+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:33.946 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:33.941+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:33.949 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:33.946+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:34.110 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:34.106+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:34.168 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:34.165+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:34.424 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:34.419+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:34.583 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:34.580+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:34.704 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:34.699+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:34.998 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:34.994+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:35.001 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:34.998+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:35.071 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:35.067+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:35.154 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:35.151+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:35.416 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:35.411+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:35.630 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:35.627+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:35.733 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:35.729+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:36.025 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:36.020+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:36.026 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:36.023+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:36.078 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:36.075+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:36.141 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:36.138+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:36.444 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:36.439+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:36.647 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:36.643+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:36.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:36.689+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:37.067 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:37.062+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:37.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:37.067+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:37.111 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:37.108+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:37.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:37.139+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:37.405 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:37.400+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:37.612 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:37.609+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:37.771 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:37.765+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:38.112 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:38.107+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:38.113 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:38.110+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:38.152 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:38.148+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:38.193 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:38.189+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:38.442 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:38.437+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:38.590 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:38.586+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:38.782 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:38.777+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:39.130 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:39.127+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:39.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:39.136+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:39.144 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:39.141+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:39.167 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:39.164+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:39.405 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:39.401+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:39.607 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:39.605+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:39.812 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:39.807+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:40.085 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:40.081+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:40.100 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:40.095+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:40.101 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:40.098+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:40.132 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:40.127+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:40.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:40.449+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:40.614 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:40.611+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:40.865 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:40.854+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:41.059 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:41.055+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:41.096 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:41.091+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:41.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:41.143+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:41.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:41.146+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:41.507 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:41.502+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:41.657 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:41.654+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:41.816 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:41.812+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:42.037 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:42.034+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:42.115 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:42.108+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:42.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:42.111+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:42.122 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:42.116+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:42.528 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:42.523+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:42.634 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:42.620+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:42.858 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:42.855+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:43.063 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:43.060+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:43.098 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:43.094+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:43.147 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:43.138+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:43.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:43.141+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:43.511 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:43.506+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:43.584 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:43.581+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:43.853 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:43.850+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:44.058 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:44.055+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:44.073 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:44.070+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:44.196 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:44.191+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:44.197 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:44.193+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:44.503 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:44.499+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:44.573 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:44.570+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:44.844 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:44.840+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:45.022 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:45.019+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:45.055 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:45.052+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:45.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:45.191+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:45.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:45.195+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:45.556 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:45.551+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:45.592 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:45.589+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:45.847 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:45.842+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:46.013 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:46.008+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:46.076 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:46.071+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:46.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:46.229+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:46.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:46.231+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:46.579 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:46.574+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:46.626 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:46.622+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:46.819 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:46.814+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:47.022 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:47.019+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:47.076 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:47.073+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:47.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:47.252+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:47.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:47.257+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:47.568 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:47.564+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:47.600 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:47.597+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:47.782 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:47.777+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:48.022 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:48.019+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:48.095 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:48.092+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:48.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:48.222+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:48.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:48.225+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:48.587 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:48.582+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:48.637 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:48.633+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:48.773 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:48.768+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:49.049 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:49.046+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:49.096 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:49.093+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:49.189 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:49.185+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:49.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:49.190+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:49.559 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:49.554+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:49.597 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:49.592+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:49.785 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:49.781+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:50.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:50.010+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:50.083 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:50.079+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:50.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:50.188+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:50.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:50.191+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:50.560 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:50.556+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:50.590 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:50.586+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:50.789 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:50.784+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:51.012 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:51.009+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:51.112 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:51.105+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:51.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:51.218+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:51.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:51.221+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:51.529 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:51.526+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:51.644 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:51.639+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:51.844 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:51.839+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:52.042 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:52.038+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:52.067 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:52.063+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:52.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:52.218+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:52.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:52.220+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:52.503 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:52.500+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:52.659 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:52.654+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:52.859 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:52.856+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:53.027 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:53.024+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:53.110 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:53.105+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:53.263 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:53.258+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:53.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:53.263+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:53.465 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:53.463+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:53.693 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:53.689+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:53.891 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:53.886+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:54.040 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:54.037+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:54.066 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:54.064+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:54.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:54.259+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:54.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:54.262+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:54.503 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:54.500+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:54.652 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:54.647+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:54.917 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:54.912+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:55.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:55.061+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:55.093 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:55.088+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:55.257 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:55.253+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:55.259 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:55.257+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:55.467 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:55.464+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:55.667 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:55.663+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:55.935 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:55.931+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:56.047 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:56.043+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:56.140 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:56.136+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:56.284 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:56.279+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:56.284 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:56.281+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:56.460 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:56.456+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:56.628 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:56.623+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:56.904 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:33:56.898+0000 1694f700 -1 osd.4 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:37.183578+0000 front 2019-12-11T23:32:37.178439+0000 (oldest deadline 2019-12-11T23:33:23.071256+0000) 2019-12-11T23:33:57.093 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:33:57.086+0000 1694f700 -1 osd.2 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.877729+0000 front 2019-12-11T23:32:41.951843+0000 (oldest deadline 2019-12-11T23:33:24.121716+0000) 2019-12-11T23:33:57.168 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:33:57.164+0000 1694f700 -1 osd.3 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:41.876040+0000 front 2019-12-11T23:32:41.787167+0000 (oldest deadline 2019-12-11T23:33:26.298816+0000) 2019-12-11T23:33:57.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:57.322+0000 1694f700 -1 osd.7 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:40.433243+0000 front 2019-12-11T23:32:40.093115+0000 (oldest deadline 2019-12-11T23:33:22.251470+0000) 2019-12-11T23:33:57.330 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:57.327+0000 1694f700 -1 osd.7 543 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:57.465 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:33:57.442+0000 1694f700 -1 osd.1 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:39.506057+0000 front 2019-12-11T23:32:39.288629+0000 (oldest deadline 2019-12-11T23:33:25.178104+0000) 2019-12-11T23:33:57.591 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:33:57.586+0000 1694f700 -1 osd.6 543 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:32:42.115456+0000 front 2019-12-11T23:32:42.078876+0000 (oldest deadline 2019-12-11T23:33:25.229802+0000) 2019-12-11T23:33:57.787 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:33:57.782+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 47sec e543 v543) 2019-12-11T23:33:58.716 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:58.570+0000 1694f700 -1 osd.7 544 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:33:59.658 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:33:59.652+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:00.786 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:00.783+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:01.811 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:01.807+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:02.779 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:02.776+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:03.739 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:03.736+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:04.764 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:04.761+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:05.737 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:05.733+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:06.736 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:06.732+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:07.738 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:07.735+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:08.705 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:08.702+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:09.660 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:09.657+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:10.695 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:10.693+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:11.730 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:11.728+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:12.731 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:12.729+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:13.709 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:13.707+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:14.693 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:14.691+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:15.739 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:15.737+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:16.775 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:16.773+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:17.757 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:17.755+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:18.793 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:18.790+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:19.819 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:19.815+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:20.872 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:20.869+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:21.908 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:21.904+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:22.882 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:22.880+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:23.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:23.848+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:24.898 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:24.895+0000 1694f700 -1 osd.7 545 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:25.921 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:25.918+0000 1694f700 -1 osd.7 546 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:26.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:26.922+0000 1694f700 -1 osd.7 546 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:27.911 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:27.907+0000 1694f700 -1 osd.7 546 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:28.960 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:28.957+0000 1694f700 -1 osd.7 546 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:29.916 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:29.913+0000 1694f700 -1 osd.7 546 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:30.936 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:30.933+0000 1694f700 -1 osd.7 546 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:32.089 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:32.038+0000 1694f700 -1 osd.7 547 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:33.017 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:33.014+0000 1694f700 -1 osd.7 548 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:34.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:34.050+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:34.650 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:34:34.643+0000 31786700 -1 osd.0 549 build_incremental_map_msg missing incremental map 549 2019-12-11T23:34:34.949 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:34:34.843+0000 32f89700 -1 osd.0 549 build_incremental_map_msg missing incremental map 549 2019-12-11T23:34:35.071 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:34:34.848+0000 32f89700 -1 osd.0 549 build_incremental_map_msg missing incremental map 549 2019-12-11T23:34:35.081 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:34:34.886+0000 31f87700 -1 osd.0 549 build_incremental_map_msg missing incremental map 549 2019-12-11T23:34:35.193 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:35.177+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:36.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:36.187+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:36.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:34:36.679+0000 32f89700 -1 osd.0 549 build_incremental_map_msg missing incremental map 549 2019-12-11T23:34:36.797 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:34:36.789+0000 11caf700 -1 osd.0 549 build_incremental_map_msg missing incremental map 549 2019-12-11T23:34:37.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:37.171+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:38.199 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:38.196+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:39.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:39.242+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:40.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:40.247+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:41.257 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:41.250+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:42.236 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:42.233+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:43.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:43.219+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:44.259 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:44.256+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:45.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:45.241+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:46.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:46.274+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:47.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:47.235+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:48.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:48.235+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:49.205 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:49.202+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:50.235 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:50.230+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:51.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:51.270+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:52.230 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:52.224+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:53.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:53.247+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:54.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:54.221+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:55.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:55.191+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:55.981 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:34:55.978+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:34:56.218 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:56.215+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:56.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:34:56.972+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:34:57.172 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:34:57.167+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:34:57.187 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:57.183+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:34:57.188 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:57.185+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:57.255 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:34:57.252+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:34:57.983 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:34:57.978+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:34:58.162 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:58.156+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:34:58.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:58.159+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:58.202 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:34:58.198+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:34:58.247 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:34:58.245+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:34:58.969 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:34:58.965+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:34:59.124 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:59.119+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:34:59.127 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:34:59.124+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:34:59.180 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:34:59.176+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:34:59.294 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:34:59.290+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:34:59.956 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:34:59.952+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:00.083 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:00.079+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:00.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:00.115+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:00.121 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:00.119+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:00.177 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:00.172+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:00.313 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:00.310+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:00.997 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:00.993+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:01.103 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:01.099+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:01.106 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:01.103+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:01.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:01.109+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:01.149 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:01.145+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:01.293 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:01.290+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:01.318 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:01.316+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:01.985 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:01.981+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:02.098 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:02.093+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:02.109 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:02.104+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:02.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:02.112+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:02.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:02.114+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:02.293 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:02.290+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:02.295 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:02.293+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:02.972 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:02.967+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:03.079 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:03.075+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:03.105 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:03.102+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:03.129 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:03.125+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:03.130 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:03.127+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:03.269 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:03.266+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:03.289 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:03.287+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:03.964 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:03.960+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:04.048 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:04.045+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:04.079 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:04.077+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:04.099 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:04.094+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:04.099 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:04.096+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:04.286 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:04.283+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:04.312 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:04.310+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:04.938 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:04.933+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:05.074 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:05.070+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:05.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:05.081+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:05.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:05.083+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:05.106 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:05.103+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:05.282 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:05.278+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:05.303 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:05.301+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:05.985 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:05.981+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:06.062 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:06.058+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:06.103 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:06.100+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:06.105 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:06.100+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:06.105 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:06.102+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:06.294 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:06.291+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:06.308 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:06.305+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:07.028 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:07.023+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:07.031 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:07.027+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:07.088 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:07.082+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:07.112 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:07.107+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:07.118 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:07.109+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:07.315 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:07.313+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:07.318 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:07.312+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:07.993 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:07.990+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:08.066 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:08.063+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:08.126 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:08.123+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:08.134 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:08.130+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:08.135 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:08.132+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:08.269 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:08.266+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:08.319 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:08.317+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:09.041 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:09.036+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:09.082 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:09.078+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:09.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:09.106+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:09.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:09.117+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:09.122 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:09.120+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:09.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:09.314+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:09.320 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:09.317+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:09.333 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:09.324+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:10.014 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:10.012+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:10.104 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:10.102+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:10.141 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:10.135+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:10.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:10.139+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:10.144 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:10.143+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:10.346 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:10.343+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:10.347 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:10.344+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:10.359 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:10.356+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:11.012 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:11.009+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:11.109 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:11.106+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:11.141 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:11.137+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:11.160 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:11.157+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:11.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:11.161+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:11.394 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:11.391+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:11.394 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:11.392+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:11.402 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:11.400+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:11.969 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:11.965+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:12.118 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:12.114+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:12.138 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:12.131+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:12.203 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:12.200+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:12.204 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:12.201+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:12.360 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:12.357+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:12.430 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:12.428+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:12.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:12.430+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:12.992 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:12.989+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:13.164 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:13.161+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:13.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:13.180+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:13.239 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:13.236+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:13.240 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:13.238+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:13.361 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:13.357+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:13.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:13.452+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:13.458 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:13.456+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:13.954 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:13.950+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:14.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:14.172+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:14.198 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:14.194+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:14.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:14.249+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:14.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:14.250+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:14.367 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:14.363+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:14.486 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:14.483+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:14.487 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:14.484+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:14.951 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:14.949+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:15.132 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:15.129+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:15.220 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:15.217+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:15.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:15.264+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:15.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:15.266+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:15.346 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:15.343+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:15.463 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:15.460+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:15.464 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:15.461+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:15.967 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:15.963+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:16.103 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:16.100+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:16.275 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:16.271+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:16.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:16.284+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:16.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:16.286+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:16.345 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:16.342+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:16.491 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:16.488+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:16.492 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:16.490+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:16.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:16.973+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:17.072 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:17.069+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:17.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:17.275+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:17.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:17.277+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:17.301 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:17.297+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:17.306 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:17.303+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:17.459 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:17.455+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:17.459 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:17.456+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:17.952 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:17.948+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:18.038 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:18.034+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:18.278 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:18.275+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:18.301 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:18.299+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:18.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:18.319+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:18.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:18.321+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:18.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:18.411+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:18.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:18.413+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:18.915 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:18.910+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:19.073 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:19.069+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:19.264 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:19.261+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:19.278 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:19.275+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:19.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:19.282+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:19.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:19.284+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:19.466 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:19.463+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:19.467 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:19.464+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:19.877 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:19.873+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:20.082 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:20.079+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:20.243 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:20.241+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:20.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:20.276+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:20.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:20.281+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:20.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:20.307+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:20.450 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:20.448+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:20.451 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:20.449+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:20.869 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:20.865+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:21.111 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:21.107+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:21.278 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:21.275+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:21.320 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:21.317+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:21.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:21.316+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:21.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:21.318+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:21.485 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:21.482+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:21.486 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:21.484+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:21.901 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:21.897+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:22.163 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:22.159+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:22.246 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:22.243+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:22.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:22.277+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:22.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:22.281+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:22.319 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:22.315+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:22.536 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:22.534+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:22.538 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:22.536+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:22.885 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:22.880+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:23.158 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:23.155+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:23.229 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:23.227+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:23.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:23.261+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:23.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:23.262+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:23.358 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:23.352+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:23.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:23.570+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:23.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:23.571+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:23.920 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:23.916+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:24.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:24.169+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:24.246 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:24.242+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:24.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:24.305+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:24.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:24.308+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:24.334 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:24.331+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:24.562 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:24.548+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:24.562 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:24.549+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:24.877 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:24.872+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:25.202 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:25.191+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:25.291 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:25.285+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:25.292 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:25.287+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:25.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:25.291+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:25.372 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:25.369+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:25.541 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:25.535+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:25.542 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:25.537+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:25.834 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:25.830+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:26.151 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:26.148+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:26.242 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:26.238+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:26.282 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:26.278+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:26.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:26.280+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:26.327 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:26.323+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:26.556 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:26.550+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:26.556 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:26.553+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:26.844 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:26.840+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:27.207 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:27.203+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:27.218 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:27.213+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:27.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:27.293+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:27.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:27.295+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:27.339 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:27.336+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:27.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:27.570+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:27.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:27.571+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:27.846 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:27.842+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:28.023 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:35:28.018+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e549 v549) 2019-12-11T23:35:28.256 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:28.253+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:28.267 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:28.264+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:28.330 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:28.326+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:28.333 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:28.330+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:28.352 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:28.347+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:28.542 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:28.537+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:28.542 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:28.539+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:28.846 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:28.840+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:29.275 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:29.272+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:29.279 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:29.276+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:29.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:29.344+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:29.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:29.348+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:29.358 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:29.352+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:29.593 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:29.588+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:29.594 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:29.590+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:29.885 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:29.882+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:30.230 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:30.227+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:30.308 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:30.304+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:30.346 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:30.341+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:30.346 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:30.343+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:30.382 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:30.379+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:30.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:30.596+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:30.600 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:30.597+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:30.893 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:30.889+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:31.229 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:31.226+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:31.331 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:31.327+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:31.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:31.329+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:31.361 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:31.356+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:31.377 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:31.374+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:31.600 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:31.597+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:31.601 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:31.598+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:31.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:31.932+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:32.246 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:32.243+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:32.318 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:32.314+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:32.330 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:32.326+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:32.331 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:32.328+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:32.411 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:32.408+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:32.646 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:32.643+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:32.647 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:32.645+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:32.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:32.971+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:33.035 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:35:33.032+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e549 v549) 2019-12-11T23:35:33.263 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:33.261+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:33.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:33.298+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:33.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:33.300+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:33.359 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:33.351+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:33.387 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:33.385+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:33.615 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:33.608+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:33.616 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:33.610+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:34.030 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:34.026+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:34.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:34.291+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:34.296 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:34.293+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:34.303 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:34.300+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:34.313 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:34.310+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:34.423 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:34.420+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:34.662 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:34.659+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:34.665 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:34.662+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:35.004 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:35.000+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:35.319 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:35.313+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:35.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:35.316+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:35.322 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:35.317+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:35.330 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:35.326+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:35.399 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:35.396+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:35.647 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:35.642+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:35.649 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:35.644+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:36.017 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:36.014+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:36.281 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:36.277+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:36.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:36.315+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:36.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:36.317+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:36.363 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:36.360+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:36.409 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:36.405+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:36.598 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:36.595+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:36.601 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:36.599+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:37.002 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:36.997+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:37.302 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:37.298+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:37.334 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:37.331+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:37.338 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:37.334+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:37.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:37.336+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:37.435 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:37.431+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:37.587 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:37.583+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:37.589 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:37.587+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:37.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:37.970+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:38.058 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:35:38.056+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e549 v549) 2019-12-11T23:35:38.257 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:38.253+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:38.339 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:38.336+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:38.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:38.370+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:38.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:38.374+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:38.442 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:38.439+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:38.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:38.571+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:38.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:38.572+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:38.951 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:38.947+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:39.217 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:39.212+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:39.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:39.290+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:39.410 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:39.407+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:39.421 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:39.418+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:39.422 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:39.419+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:39.600 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:39.597+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:39.600 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:39.598+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:39.991 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:39.987+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:40.210 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:40.206+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:40.248 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:40.244+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:40.446 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:40.441+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:40.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:40.456+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:40.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:40.458+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:40.605 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:40.603+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:40.606 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:40.604+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:40.989 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:40.984+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:41.233 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:41.229+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:41.283 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:41.279+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:41.475 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:41.472+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:41.511 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:41.507+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:41.512 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:41.509+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:41.661 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:41.656+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:41.661 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:41.658+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:42.020 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:42.016+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:42.201 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:42.197+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:42.259 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:35:42.256+0000 1694f700 -1 osd.2 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.811187+0000 front 2019-12-11T23:34:13.863183+0000 (oldest deadline 2019-12-11T23:34:59.636440+0000) 2019-12-11T23:35:42.433 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:35:42.430+0000 1694f700 -1 osd.1 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.965049+0000 front 2019-12-11T23:34:13.958599+0000 (oldest deadline 2019-12-11T23:34:56.237398+0000) 2019-12-11T23:35:42.542 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:42.539+0000 1694f700 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:14.925855+0000 front 2019-12-11T23:34:14.880675+0000 (oldest deadline 2019-12-11T23:34:57.064029+0000) 2019-12-11T23:35:42.545 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:42.543+0000 1694f700 -1 osd.7 549 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:42.696 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:42.692+0000 1694f700 -1 osd.0 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:34:28.338908+0000 (oldest deadline 2019-12-11T23:35:08.338908+0000) 2019-12-11T23:35:42.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:42.696+0000 1694f700 -1 osd.0 549 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:43.028 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:35:43.024+0000 1694f700 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:10.175528+0000 front 2019-12-11T23:34:10.267006+0000 (oldest deadline 2019-12-11T23:34:55.314129+0000) 2019-12-11T23:35:43.132 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:35:43.125+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e549 v549) 2019-12-11T23:35:43.240 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:35:43.218+0000 1694f700 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:34:13.980346+0000 front 2019-12-11T23:34:14.004653+0000 (oldest deadline 2019-12-11T23:34:56.851110+0000) 2019-12-11T23:35:43.997 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:35:43.802+0000 1694f700 -1 osd.7 550 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:117826 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e543) 2019-12-11T23:35:44.050 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:43.899+0000 1694f700 -1 osd.0 550 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e547) 2019-12-11T23:35:44.138 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:35:44.121+0000 34f8d700 -1 osd.0 550 build_incremental_map_msg missing incremental map 549 2019-12-11T23:36:44.567 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:04:31.500 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.574 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:04:31.507 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.587 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:04:31.519 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.592 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:04:31.496 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.592 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:04:31.524 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.617 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:04:31.521 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.884 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:04:31.788 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.888 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:04:31.821 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:36:44.908 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:04:31.841 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:03.622 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:03.620+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:04.443 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:04.440+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:04.614 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:04.611+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:05.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:05.424+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:05.624 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:05.621+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:06.388 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:06.386+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:06.633 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:06.631+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:07.353 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:07.351+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:07.611 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:07.609+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:08.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:08.309+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:08.597 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:08.595+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:09.300 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:09.299+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:09.619 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:09.617+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:10.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:10.348+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:10.667 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:10.663+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:10.668 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:10.665+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:11.013 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:11.009+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:11.306 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:11.305+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:11.658 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:11.653+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:11.659 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:11.657+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:12.007 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:12.003+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:12.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:12.284+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:12.666 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:12.661+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:12.668 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:12.666+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:13.036 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:13.033+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:13.057 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:13.054+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:13.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:13.264+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:13.720 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:13.716+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:13.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:13.719+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:14.051 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:14.046+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:14.078 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:14.076+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:14.222 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:14.216+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:14.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:14.219+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:14.346 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:14.343+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:14.712 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:14.708+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:14.713 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:14.710+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:14.783 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:14.779+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:15.041 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:15.036+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:15.124 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:15.120+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:15.225 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:15.221+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:15.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:15.226+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:15.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:15.380+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:15.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:15.717+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:15.726 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:15.720+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:15.746 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:15.742+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:16.051 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:16.047+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:16.167 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:16.164+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:16.186 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:16.181+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:16.188 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:16.186+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:16.417 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:16.414+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:16.727 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:16.724+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:16.772 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:16.767+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:16.772 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:16.770+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:17.009 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:17.005+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:17.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:17.209+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:17.232 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:17.226+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:17.233 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:17.229+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:17.402 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:17.399+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:17.742 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:17.738+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:17.807 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:17.802+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:17.807 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:17.804+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:17.972 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:17.968+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:18.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:18.209+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:18.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:18.239+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:18.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:18.242+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:18.377 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:18.374+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:18.775 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:18.770+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:18.776 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:18.773+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:18.787 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:18.784+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:18.951 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:18.946+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:19.214 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:19.211+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:19.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:19.283+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:19.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:19.286+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:19.394 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:19.391+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:19.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:19.782+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:19.829 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:19.818+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:19.829 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:19.820+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:19.961 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:19.957+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:20.217 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:20.214+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:20.255 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:20.252+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:20.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:20.254+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:20.353 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:20.349+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:20.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:20.785+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:20.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:20.787+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:20.812 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:20.808+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:20.929 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:20.925+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:21.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:21.228+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:21.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:21.264+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:21.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:21.267+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:21.356 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:21.352+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:21.773 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:21.768+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:21.775 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:21.772+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:21.821 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:21.818+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:21.947 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:21.943+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:22.202 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:22.199+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:22.253 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:22.249+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:22.254 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:22.251+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:22.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:22.381+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:22.737 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:22.732+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:22.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:22.735+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:22.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:22.777+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:22.905 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:22.900+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:23.168 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:23.164+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:23.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:23.284+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:23.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:23.287+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:23.408 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:23.404+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:23.711 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:23.706+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:23.712 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:23.709+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:23.722 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:05:10.654 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.751 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:05:10.684 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.791 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:05:10.723 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.792 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:05:10.724 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.809 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:23.807+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:23.824 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:05:10.729 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.827 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:05:10.731 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.833 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:05:10.766 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.855 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:05:10.787 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.905 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:05:10.809 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:37:23.937 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:23.932+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:24.217 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:24.213+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:24.275 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:24.270+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:24.276 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:24.273+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:24.442 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:24.440+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:24.729 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:24.724+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:24.730 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:24.727+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:24.788 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:24.785+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:24.971 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:24.967+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:25.197 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:25.194+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:25.291 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:25.288+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:25.291 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:25.289+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:25.434 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:25.431+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:25.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:25.734+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:25.741 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:25.738+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:25.766 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:25.762+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:26.003 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:25.999+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:26.157 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:26.154+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:26.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:26.243+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:26.249 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:26.246+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:26.415 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:26.413+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:26.781 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:26.777+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:26.785 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:26.782+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:26.796 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:26.793+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:26.958 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:26.954+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:27.158 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:27.154+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:27.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:27.273+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:27.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:27.275+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:27.432 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:27.428+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:27.759 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:27.754+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:27.760 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:27.757+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:27.836 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:27.833+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:27.925 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:27.920+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:28.160 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:28.157+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:28.276 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:28.269+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:28.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:28.272+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:28.465 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:28.461+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:28.742 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:28.738+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:28.745 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:28.742+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:28.816 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:28.813+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:28.927 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:28.922+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:29.209 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:29.206+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:29.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:29.322+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:29.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:29.324+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:29.436 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:29.433+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:29.766 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:29.761+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:29.767 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:29.764+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:29.825 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:29.822+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:29.948 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:29.942+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:30.201 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:30.199+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:30.374 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:30.368+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:30.375 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:30.370+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:30.456 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:30.453+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:30.735 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:30.732+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:30.737 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:30.735+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:30.797 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:30.795+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:30.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:30.931+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:31.247 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:31.243+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:31.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:31.387+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:31.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:31.390+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:31.480 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:31.477+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:31.748 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:31.744+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:31.749 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:31.746+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:31.766 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:31.763+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:31.977 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:31.973+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:32.271 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:32.267+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:32.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:32.424+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:32.438 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:32.436+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:32.477 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:32.474+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:32.740 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:32.735+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:32.770 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:32.765+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:32.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:32.768+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:32.995 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:32.990+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:33.261 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:33.257+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:33.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:33.445+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:33.450 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:33.447+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:33.509 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:33.507+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:33.782 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:33.779+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:33.818 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:33.814+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:33.821 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:33.818+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:33.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:33.994+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:34.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:34.229+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:34.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:34.441+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:34.448 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:34.444+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:34.494 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:34.490+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:34.765 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:34.762+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:34.774 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:34.770+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:34.778 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:34.775+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:35.051 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:35.046+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:35.199 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:35.194+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:35.456 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:35.451+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:35.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:35.479+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:35.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:35.482+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:35.739 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:35.735+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:35.779 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:35.774+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:35.781 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:35.779+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:36.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:36.033+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:36.231 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:36.227+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:36.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:36.470+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:36.523 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:36.515+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:36.526 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:36.519+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:36.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:36.782+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:36.811 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:36.806+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:36.811 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:36.808+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:37.000 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:36.995+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:37.230 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:37.225+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:37.498 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:37.493+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:37.522 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:37.518+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:37.523 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:37.520+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:37.805 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:37.790+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:37.830 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:37.825+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:37.830 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:37.827+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:37.970 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:37.965+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:38.195 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:38.191+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:38.465 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:38.460+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:38.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:38.548+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:38.555 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:38.551+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:38.790 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:38.786+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:38.847 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:38.842+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:38.848 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:38.844+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:38.928 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:38.925+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:39.216 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:39.212+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:39.468 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:39.462+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:39.532 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:39.527+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:39.536 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:39.530+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:39.753 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:39.748+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:39.889 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:39.884+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:39.890 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:39.885+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:39.891 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:39.888+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:40.227 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:40.222+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:40.475 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:40.470+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:40.491 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:40.486+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:40.491 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:40.489+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:40.787 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:40.783+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:40.856 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:40.851+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:40.893 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:40.889+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:40.896 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:40.893+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:41.258 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:41.254+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:41.509 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:41.505+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:41.529 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:41.525+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:41.533 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:41.530+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:41.756 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:41.753+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:41.858 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:41.854+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:41.910 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:41.907+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:41.913 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:41.910+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:42.215 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:42.211+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:42.530 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:42.527+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:42.556 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:42.552+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:42.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:42.556+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:42.804 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:42.802+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:42.876 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:42.871+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:42.877 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:42.873+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:42.903 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:42.900+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:43.212 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:43.209+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:43.519 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:43.516+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:43.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:43.586+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:43.594 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:43.591+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:43.781 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:37:43.778+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 42sec e554 v554) 2019-12-11T23:37:43.817 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:43.813+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:43.873 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:43.868+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:43.907 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:43.902+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:43.907 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:43.904+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:44.210 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:44.206+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:44.558 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:44.554+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:44.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:44.603+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:44.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:44.607+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:44.851 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:44.845+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:44.856 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:44.852+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:44.880 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:44.875+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:44.881 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:44.878+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:45.175 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:45.171+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:45.510 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:45.507+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:45.613 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:45.609+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:45.616 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:45.614+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:45.832 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:45.829+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:45.842 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:45.838+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:45.843 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:45.840+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:45.890 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:45.886+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:46.192 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:46.189+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:46.554 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:46.551+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:46.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:46.602+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:46.608 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:46.603+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:46.796 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:46.793+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:46.879 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:46.874+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:46.879 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:46.876+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:46.906 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:46.898+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:47.243 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:47.238+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:47.510 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:47.506+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:47.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:47.601+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:47.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:47.604+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:47.842 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:47.821+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:47.865 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:47.862+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:47.872 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:47.868+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:47.873 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:47.870+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:48.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:37:48.274+0000 1694f700 -1 osd.2 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.107527+0000 front 2019-12-11T23:36:28.463088+0000 (oldest deadline 2019-12-11T23:37:12.180644+0000) 2019-12-11T23:37:48.559 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:37:48.556+0000 1694f700 -1 osd.1 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:28.931914+0000 front 2019-12-11T23:36:28.904992+0000 (oldest deadline 2019-12-11T23:37:13.580469+0000) 2019-12-11T23:37:48.624 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:48.619+0000 1694f700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.354435+0000 front 2019-12-11T23:36:29.352445+0000 (oldest deadline 2019-12-11T23:37:13.305631+0000) 2019-12-11T23:37:48.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:48.622+0000 1694f700 -1 osd.7 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:48.822 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:37:48.819+0000 1694f700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.456768+0000 front 2019-12-11T23:36:29.695712+0000 (oldest deadline 2019-12-11T23:37:14.704941+0000) 2019-12-11T23:37:48.833 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:37:48.829+0000 1694f700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:27.633778+0000 front 2019-12-11T23:36:27.625043+0000 (oldest deadline 2019-12-11T23:37:10.415301+0000) 2019-12-11T23:37:48.847 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:37:48.844+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 42sec e554 v554) 2019-12-11T23:37:48.863 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:48.859+0000 1694f700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:36:29.250214+0000 front 2019-12-11T23:36:29.261377+0000 (oldest deadline 2019-12-11T23:37:10.272761+0000) 2019-12-11T23:37:48.865 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:48.863+0000 1694f700 -1 osd.6 554 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:49.877 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:49.874+0000 1694f700 -1 osd.6 555 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:49.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:49.867+0000 1694f700 -1 osd.7 555 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:50.954 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:37:50.951+0000 1694f700 -1 osd.6 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88965 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:51.074 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:51.070+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:52.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:52.343+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:53.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:53.366+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:54.383 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:54.381+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:55.373 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:55.370+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:56.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:56.419+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:57.406 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:57.403+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:58.433 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:58.431+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:37:59.452 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:37:59.449+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:00.490 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:00.487+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:01.493 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:01.490+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:02.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:02.499+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:03.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:03.484+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:04.479 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:04.475+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:05.520 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:05.518+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:06.508 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:06.504+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:07.466 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:07.465+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:08.515 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:08.512+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:09.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:09.471+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:10.527 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:10.525+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:11.577 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:11.575+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:12.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:12.610+0000 1694f700 -1 osd.7 556 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:13.638 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:13.613+0000 1694f700 -1 osd.7 557 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:14.586 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:14.584+0000 1694f700 -1 osd.7 557 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:15.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:15.620+0000 1694f700 -1 osd.7 557 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:16.891 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:16.669+0000 1694f700 -1 osd.7 558 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:17.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:17.627+0000 1694f700 -1 osd.7 558 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:18.650 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:18.648+0000 1694f700 -1 osd.7 559 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:19.694 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:19.691+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:20.709 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:20.701+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:21.746 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:21.743+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:22.757 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:22.748+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:23.798 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:23.795+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:24.851 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:24.843+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:25.815 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:25.812+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:26.790 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:26.787+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:27.790 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:27.787+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:28.838 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:28.835+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:29.860 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:29.857+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:30.868 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:30.865+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:31.852 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:31.849+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:32.844 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:32.841+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:33.853 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:33.850+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:34.866 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:34.863+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:35.823 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:35.821+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:36.849 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:36.843+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:37.901 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:37.898+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:38.900 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:38.897+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:39.939 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:39.936+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:40.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:40.921+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:41.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:41.935+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:42.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:42.968+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:43.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:43.970+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:44.972 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:44.968+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:45.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:45.942+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:46.965 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:46.962+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:47.562 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:47.557+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:47.982 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:47.979+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:48.519 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:48.514+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:49.039 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:49.037+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:49.379 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:49.375+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:49.542 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:49.539+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:50.048 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:50.043+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:50.049 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:50.045+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:50.415 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:50.412+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:50.488 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:50.486+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:50.514 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:50.512+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:50.516 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:50.514+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:50.823 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:50.820+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:51.049 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:51.046+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:51.050 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:51.047+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:51.461 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:51.450+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:51.493 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:51.491+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:51.503 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:51.500+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:51.505 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:51.503+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:51.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:51.786+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:52.042 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:52.039+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:52.042 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:52.040+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:52.462 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:52.455+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:52.469 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:52.466+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:52.505 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:52.501+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:52.530 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:52.527+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:52.758 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:52.747+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:53.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:53.040+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:53.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:53.042+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:53.456 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:53.453+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:53.480 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:53.477+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:53.518 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:53.513+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:53.582 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:53.578+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:53.766 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:53.762+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:54.057 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:54.053+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:54.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:54.055+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:54.452 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:54.449+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:54.493 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:54.491+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:54.536 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:54.532+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:54.578 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:54.574+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:54.779 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:54.775+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:55.041 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:55.038+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:55.046 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:55.042+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:55.416 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:55.412+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:55.487 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:55.479+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:55.570 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:55.567+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:55.620 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:55.615+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:55.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:55.766+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:56.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:56.072+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:56.078 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:56.076+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:56.382 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:56.378+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:56.503 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:56.501+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:56.611 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:56.607+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:56.615 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:56.612+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:56.799 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:56.795+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:57.093 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:57.090+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:57.097 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:57.094+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:57.409 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:57.406+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:57.541 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:57.538+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:57.585 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:57.580+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:57.599 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:57.596+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:57.787 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:57.783+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:58.076 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:58.073+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:58.079 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:58.077+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:58.375 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:58.370+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:58.564 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:58.561+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:58.592 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:58.589+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:58.595 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:58.591+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:58.841 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:58.832+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:38:59.046 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:59.042+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:38:59.047 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:38:59.044+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:38:59.382 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:38:59.380+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:38:59.551 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:38:59.549+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:38:59.564 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:38:59.560+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:38:59.616 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:38:59.614+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:38:59.858 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:38:59.854+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:00.079 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:00.076+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:00.082 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:00.080+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:00.387 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:00.385+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:00.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:00.501+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:00.574 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:00.570+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:00.666 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:00.663+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:00.875 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:00.872+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:01.049 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:01.045+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:01.052 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:01.049+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:01.377 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:01.374+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:01.480 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:01.477+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:01.560 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:01.556+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:01.619 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:01.616+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:01.848 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:01.843+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:02.007 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:02.004+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:02.009 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:02.007+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:02.405 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:02.402+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:02.467 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:02.461+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:02.601 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:02.598+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:02.630 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:02.627+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:02.818 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:02.815+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:03.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:03.015+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:03.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:03.017+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:03.411 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:03.407+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:03.493 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:03.489+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:03.588 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:03.584+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:03.681 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:03.677+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:03.782 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:03.780+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:04.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:04.067+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:04.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:04.069+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:04.447 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:04.443+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:04.543 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:04.539+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:04.600 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:04.597+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:04.681 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:04.677+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:04.801 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:04.798+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:05.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:05.069+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:05.074 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:05.073+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:05.439 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:05.435+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:05.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:05.546+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:05.574 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:05.571+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:05.654 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:05.650+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:05.789 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:05.787+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:06.121 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:06.119+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:06.123 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:06.122+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:06.479 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:06.476+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:06.572 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:06.570+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:06.597 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:06.584+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:06.637 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:06.633+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:06.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:06.787+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:07.129 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:07.127+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:07.132 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:07.130+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:07.435 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:07.430+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:07.532 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:07.527+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:07.624 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:07.620+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:07.643 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:07.640+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:07.786 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:07.784+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:08.126 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:08.123+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:08.129 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:08.124+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:08.477 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:08.474+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:08.584 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:08.581+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:08.593 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:08.585+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:08.618 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:08.614+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:08.772 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:08.769+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:09.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:09.159+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:09.164 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:09.161+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:09.460 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:09.458+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:09.596 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:09.594+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:09.610 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:09.607+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:09.636 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:09.634+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:09.809 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:09.804+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:10.161 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:10.159+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:10.162 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:10.160+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:10.468 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:10.465+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:10.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:10.605+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:10.635 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:10.633+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:10.639 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:10.634+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:10.847 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:10.842+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:11.151 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:11.146+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:11.151 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:11.148+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:11.512 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:11.510+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:11.615 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:11.611+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:11.654 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:11.651+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:11.655 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:11.651+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:11.867 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:11.864+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:12.193 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:12.189+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:12.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:12.191+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:12.489 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:12.487+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:12.622 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:12.619+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:12.652 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:12.648+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:12.659 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:12.654+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:12.915 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:12.911+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:13.166 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:13.161+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:13.166 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:13.163+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:13.526 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:13.523+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:13.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:13.577+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:13.657 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:13.654+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:13.658 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:13.655+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:13.704 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:13.700+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:13.943 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:13.939+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:14.179 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:14.175+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:14.179 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:14.177+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:14.542 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:14.539+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:14.578 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:14.575+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:14.633 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:14.624+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:14.633 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:14.627+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:14.674 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:14.670+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:14.950 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:14.946+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:15.149 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:15.146+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:15.151 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:15.149+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:15.513 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:15.510+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:15.561 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:15.558+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:15.596 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:15.592+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:15.596 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:15.594+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:15.677 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:15.671+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:15.998 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:15.995+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:16.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:16.189+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:16.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:16.193+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:16.547 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:16.543+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:16.565 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:16.562+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:16.593 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:16.590+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:16.596 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:16.594+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:16.719 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:16.715+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:17.023 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:17.019+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:17.182 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:17.178+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:17.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:17.180+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:17.575 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:17.571+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:17.617 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:17.614+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:17.636 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:17.633+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:17.637 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:17.634+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:17.729 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:17.722+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:18.077 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:18.073+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:18.187 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:18.183+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:18.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:18.187+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:18.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:18.530+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:18.625 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:18.621+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:18.637 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:18.630+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:18.638 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:18.632+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:18.729 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:18.726+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:19.032 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:19.028+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:19.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:19.177+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:19.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:19.178+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:19.204 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:39:19.197+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e560 v560) 2019-12-11T23:39:19.567 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:19.564+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:19.632 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:19.629+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:19.635 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:19.632+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:19.636 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:19.634+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:19.757 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:19.753+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:20.085 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:20.081+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:20.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:20.194+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:20.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:20.199+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:20.557 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:20.555+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:20.641 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:20.635+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:20.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:20.635+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:20.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:20.637+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:20.722 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:20.719+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:21.129 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:21.124+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:21.244 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:21.240+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:21.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:21.242+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:21.523 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:21.520+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:21.625 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:21.621+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:21.625 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:21.623+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:21.630 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:21.627+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:21.709 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:21.695+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:22.152 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:22.148+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:22.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:22.283+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:22.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:22.284+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:22.513 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:22.502+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:22.618 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:22.616+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:22.628 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:22.618+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:22.634 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:22.626+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:22.661 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:22.658+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:23.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:23.138+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:23.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:23.319+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:23.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:23.320+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:23.539 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:23.537+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:23.622 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:23.619+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:23.623 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:23.620+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:23.626 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:23.624+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:23.669 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:23.666+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:24.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:24.193+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:24.219 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:39:24.214+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e560 v560) 2019-12-11T23:39:24.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:24.315+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:24.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:24.322+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:24.590 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:24.587+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:24.607 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:24.603+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:24.671 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:24.669+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:24.672 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:24.669+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:24.673 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:24.670+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:25.248 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:25.244+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:25.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:25.346+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:25.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:25.347+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:25.542 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:25.540+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:25.605 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:25.603+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:25.630 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:25.626+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:25.630 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:25.628+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:25.688 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:25.684+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:26.277 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:26.272+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:26.376 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:26.372+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:26.379 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:26.377+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:26.569 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:26.564+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:26.583 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:26.579+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:26.607 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:26.604+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:26.608 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:26.605+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:26.677 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:26.673+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:27.308 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:27.304+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:27.393 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:27.384+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:27.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:27.388+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:27.547 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:27.545+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:27.570 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:27.568+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:27.570 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:27.569+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:27.584 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:27.581+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:27.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:27.693+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:28.292 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:28.289+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:28.359 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:28.356+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:28.362 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:28.360+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:28.523 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:28.520+0000 1694f700 -1 osd.0 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:38:14.423773+0000 (oldest deadline 2019-12-11T23:39:13.266808+0000) 2019-12-11T23:39:28.523 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:28.521+0000 1694f700 -1 osd.0 560 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:39:28.536 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:39:28.531+0000 1694f700 -1 osd.2 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:02.171905+0000 front 2019-12-11T23:38:02.208482+0000 (oldest deadline 2019-12-11T23:38:47.437022+0000) 2019-12-11T23:39:28.597 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:39:28.591+0000 1694f700 -1 osd.1 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:04.878132+0000 front 2019-12-11T23:38:04.681405+0000 (oldest deadline 2019-12-11T23:38:49.868333+0000) 2019-12-11T23:39:28.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:39:28.691+0000 1694f700 -1 osd.4 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.448101+0000 front 2019-12-11T23:38:05.620140+0000 (oldest deadline 2019-12-11T23:38:48.884413+0000) 2019-12-11T23:39:29.253 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:39:29.249+0000 1694f700 -1 osd.6 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:05.724041+0000 front 2019-12-11T23:38:05.718775+0000 (oldest deadline 2019-12-11T23:38:50.376899+0000) 2019-12-11T23:39:29.284 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:39:29.281+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e560 v560) 2019-12-11T23:39:29.384 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:29.365+0000 1694f700 -1 osd.7 560 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:38:06.434631+0000 front 2019-12-11T23:38:06.491996+0000 (oldest deadline 2019-12-11T23:38:49.853590+0000) 2019-12-11T23:39:29.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:39:29.367+0000 1694f700 -1 osd.7 560 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118185 2.2fs0 2:f6be2fc9:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%e5%98%87%e3%9c%84%e3%9c%86%e5%98%87%e1%a0%83%e1%a0%82%e1%a0%81%e5%98%86%ef%84%91%e0%a4%82%e9%90%89%e5%98%88%e9%90%8b%ed%88%8d%e5%98%85:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e554) 2019-12-11T23:39:30.127 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:39:30.024+0000 1694f700 -1 osd.0 561 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e558) 2019-12-11T23:40:01.962 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:40:01.779+0000 3578e700 -1 osd.1 564 build_incremental_map_msg missing incremental map 564 2019-12-11T23:40:01.966 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:40:01.782+0000 35f8f700 -1 osd.1 564 build_incremental_map_msg missing incremental map 564 2019-12-11T23:40:02.026 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:40:01.821+0000 31f87700 -1 osd.1 564 build_incremental_map_msg missing incremental map 564 2019-12-11T23:40:02.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:40:01.862+0000 3578e700 -1 osd.1 564 build_incremental_map_msg missing incremental map 564 2019-12-11T23:40:02.162 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:40:01.956+0000 31f87700 -1 osd.1 564 build_incremental_map_msg missing incremental map 564 2019-12-11T23:40:02.214 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:40:01.991+0000 32788700 -1 osd.1 564 build_incremental_map_msg missing incremental map 564 2019-12-11T23:40:02.350 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:40:02.320+0000 34f8d700 -1 osd.1 564 build_incremental_map_msg missing incremental map 564 2019-12-11T23:40:04.485 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:07:51.418 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.519 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:07:51.452 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.525 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:07:51.458 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.526 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:07:51.459 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.548 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:07:51.453 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.551 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:07:51.455 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.696 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:07:51.629 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.708 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:07:51.641 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:04.842 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:07:51.746 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.721 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:08:17.648 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.721 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:08:17.653 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.737 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:08:17.663 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.741 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:08:17.670 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.745 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:08:17.650 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.759 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:08:17.664 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.777 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:08:17.710 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.797 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:08:17.729 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:40:30.862 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:08:17.767 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:41:02.899 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:02.896+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:02.901 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:02.898+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:03.137 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:03.133+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:03.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:03.319+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:03.908 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:03.905+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:03.924 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:03.920+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:04.175 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:04.172+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:04.373 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:04.368+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:04.885 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:04.882+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:04.966 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:04.961+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:05.173 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:05.169+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:05.415 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:05.410+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:05.455 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:05.453+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:05.860 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:05.858+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:05.933 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:05.929+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:06.191 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:06.188+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:06.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:06.386+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:06.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:06.474+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:06.867 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:06.864+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:06.955 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:06.953+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:07.200 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:07.198+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:07.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:07.420+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:07.426 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:07.423+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:07.444 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:07.441+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:07.827 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:07.824+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:08.000 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:07.993+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:08.179 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:08.176+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:08.182 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:08.178+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:08.404 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:08.400+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:08.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:08.467+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:08.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:08.470+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:08.779 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:08.777+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:08.985 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:08.981+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:09.149 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:09.143+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:09.187 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:09.182+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:09.453 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:09.450+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:09.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:09.495+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:09.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:09.498+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:09.745 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:09.741+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:09.961 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:09.957+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:10.133 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:10.129+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:10.183 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:10.180+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:10.449 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:10.445+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:10.458 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:10.453+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:10.458 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:10.456+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:10.704 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:10.701+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:10.922 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:10.915+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:11.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:11.086+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:11.160 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:11.156+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:11.438 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:11.434+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:11.439 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:11.436+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:11.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:11.473+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:11.696 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:11.692+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:11.966 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:11.961+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:12.076 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:12.072+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:12.196 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:12.192+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:12.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:12.440+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:12.447 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:12.442+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:12.495 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:12.491+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:12.656 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:12.652+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:12.996 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:12.990+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:13.072 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:13.069+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:13.214 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:13.210+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:13.454 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:13.449+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:13.462 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:13.457+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:13.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:13.460+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:13.637 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:13.631+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:14.003 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:13.999+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:14.025 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:14.022+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:14.254 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:14.250+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:14.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:14.449+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:14.456 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:14.451+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:14.458 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:14.454+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:14.643 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:14.637+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:15.054 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:15.050+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:15.064 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:15.061+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:15.275 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:15.257+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:15.451 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:15.447+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:15.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:15.466+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:15.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:15.468+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:15.684 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:15.680+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:16.035 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:16.031+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:16.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:16.086+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:16.286 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:16.282+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:16.454 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:16.449+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:16.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:16.452+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:16.483 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:16.478+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:16.695 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:16.691+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:17.022 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:17.017+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:17.062 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:17.058+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:17.313 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:17.309+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:17.461 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:17.456+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:17.462 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:17.459+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:17.481 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:17.476+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:17.734 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:17.729+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:18.034 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:18.030+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:18.092 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:18.088+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:18.281 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:18.277+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:18.487 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:18.481+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:18.488 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:18.485+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:18.494 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:18.490+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:18.713 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:18.708+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:19.029 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:19.024+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:19.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:19.128+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:19.275 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:19.271+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:19.506 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:19.502+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:19.519 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:19.514+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:19.520 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:19.517+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:19.677 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:19.673+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:19.999 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:19.996+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:20.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:20.082+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:20.285 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:20.280+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:20.507 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:20.503+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:20.562 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:20.558+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:20.563 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:20.560+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:20.725 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:20.722+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:21.030 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:21.025+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:21.125 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:21.121+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:21.253 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:21.249+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:21.470 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:21.467+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:21.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:21.555+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:21.561 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:21.558+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:21.721 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:21.718+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:22.040 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:22.035+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:22.080 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:22.077+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:22.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:22.219+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:22.507 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:22.505+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:22.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:22.599+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:22.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:22.604+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:22.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:22.753+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:23.076 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:23.071+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:23.108 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:23.106+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:23.213 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:23.206+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:23.532 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:23.528+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:23.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:23.645+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:23.650 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:23.647+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:23.756 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:23.753+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:24.038 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:24.035+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:24.135 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:24.133+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:24.176 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:24.170+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:24.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:24.489+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:24.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:24.626+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:24.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:24.628+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:24.807 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:24.804+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:25.056 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:25.052+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:25.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:25.105+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:25.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:25.151+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:25.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:25.464+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:25.657 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:25.652+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:25.658 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:25.655+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:25.795 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:25.792+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:26.055 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:26.051+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:26.124 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:26.120+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:26.154 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:26.152+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:26.508 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:26.505+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:26.629 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:26.625+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:26.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:26.627+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:26.845 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:26.841+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:27.074 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:27.071+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:27.114 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:27.110+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:27.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:27.124+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:27.506 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:27.502+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:27.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:27.615+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:27.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:27.617+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:27.845 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:27.841+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:28.093 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:28.089+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:28.125 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:28.121+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:28.130 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:28.127+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:28.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:28.530+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:28.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:28.579+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:28.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:28.582+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:28.890 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:28.886+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:29.094 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:29.090+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:29.104 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:29.100+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:29.145 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:29.141+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:29.504 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:29.501+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:29.636 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:29.631+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:29.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:29.634+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:29.930 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:29.926+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:30.104 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:30.100+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:30.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:30.128+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:30.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:30.138+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:30.556 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:30.552+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:30.634 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:30.631+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:30.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:30.634+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:30.914 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:30.911+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:31.140 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:31.136+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:31.147 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:31.144+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:31.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:31.191+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:31.600 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:31.597+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:31.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:31.626+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:31.632 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:31.628+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:31.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:31.899+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:32.163 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:32.159+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:32.181 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:32.178+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:32.209 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:32.204+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:32.572 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:32.569+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:32.606 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:32.602+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:32.609 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:32.607+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:32.936 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:32.933+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:33.190 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:33.187+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:33.194 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:33.190+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:33.261 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:33.257+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:33.551 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:33.547+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:33.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:33.615+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:33.623 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:33.618+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:33.972 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:33.968+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:34.159 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:34.155+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:34.189 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:34.185+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:34.292 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:34.286+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:34.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:34.530+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:34.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:34.667+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:34.673 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:34.670+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:34.959 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:34.955+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:35.119 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:35.115+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:35.143 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:35.139+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:35.301 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:35.297+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:35.518 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:35.515+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:35.667 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:35.661+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:35.667 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:35.663+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:36.000 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:35.995+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:36.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:36.123+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:36.173 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:36.170+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:36.306 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:36.303+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:36.535 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:36.531+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:36.654 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:36.649+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:36.655 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:36.652+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:36.995 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:36.991+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:37.116 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:37.113+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:37.153 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:37.149+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:37.305 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:37.302+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:37.529 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:37.525+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:37.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:37.613+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:37.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:37.616+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:37.966 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:37.963+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:38.116 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:38.113+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:38.147 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:38.144+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:38.302 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:38.298+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:38.540 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:38.536+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:38.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:38.644+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:38.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:38.650+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:38.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:41:38.940+0000 1694f700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.171762+0000 front 2019-12-11T23:40:18.728640+0000 (oldest deadline 2019-12-11T23:41:02.851881+0000) 2019-12-11T23:41:39.101 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:41:39.098+0000 1694f700 -1 osd.1 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:20.265171+0000 front 2019-12-11T23:40:20.266390+0000 (oldest deadline 2019-12-11T23:41:02.521621+0000) 2019-12-11T23:41:39.190 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:41:39.186+0000 1694f700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:18.962501+0000 front 2019-12-11T23:40:19.204927+0000 (oldest deadline 2019-12-11T23:41:07.558557+0000) 2019-12-11T23:41:39.259 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:41:39.254+0000 1694f700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:19.876687+0000 front 2019-12-11T23:40:19.921773+0000 (oldest deadline 2019-12-11T23:41:02.134110+0000) 2019-12-11T23:41:39.592 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:41:39.588+0000 1694f700 -1 osd.2 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.250518+0000 front 2019-12-11T23:40:21.440389+0000 (oldest deadline 2019-12-11T23:41:05.038548+0000) 2019-12-11T23:41:39.695 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:39.692+0000 1694f700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:40:21.994175+0000 front 2019-12-11T23:40:21.981973+0000 (oldest deadline 2019-12-11T23:41:03.015913+0000) 2019-12-11T23:41:39.699 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:39.696+0000 1694f700 -1 osd.7 565 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:39.804 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:41:39.799+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 45sec e565 v565) 2019-12-11T23:41:41.030 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:41.005+0000 1694f700 -1 osd.7 566 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:42.020 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:42.019+0000 1694f700 -1 osd.7 567 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:41:43.130 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:41:42.989+0000 1694f700 -1 osd.7 567 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:118640 2.27s0 2:e70e32de:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.672_%ef%84%91%e3%9c%85%e8%94%8b%eb%8c%8e%e3%9c%84%e5%98%87%ef%84%8f%e1%a0%82%ef%84%8f%ed%88%8e%ce%a9%eb%8c%8c%e5%98%85%e3%9c%84%e8%94%89:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e565) 2019-12-11T23:42:38.955 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:38.950+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:39.951 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:39.948+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:40.132 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:40.128+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:40.998 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:40.996+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:41.017 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:41.012+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:41.180 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:41.176+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:41.324 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:41.320+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:41.520 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:41.517+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:42.011 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:42.006+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:42.012 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:42.009+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:42.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:42.161+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:42.317 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:42.314+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:42.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:42.498+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:43.003 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:42.999+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:43.054 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:43.050+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:43.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:43.210+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:43.291 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:43.288+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:43.485 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:43.481+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:43.567 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:43.563+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:44.026 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:44.024+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:44.065 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:44.060+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:44.181 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:44.177+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:44.332 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:44.329+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:44.468 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:44.462+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:44.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:44.571+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:44.999 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:44.995+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:45.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:45.086+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:45.206 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:45.202+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:45.330 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:45.325+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:45.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:45.503+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:45.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:45.535+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:45.993 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:45.990+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:46.097 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:46.092+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:46.164 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:46.161+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:46.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:46.346+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:46.480 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:46.478+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:46.527 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:46.524+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:46.954 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:46.947+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:47.106 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:47.103+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:47.171 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:47.165+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:47.382 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:47.379+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:47.485 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:47.482+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:47.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:47.561+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:47.938 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:47.934+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:48.130 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:48.126+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:48.137 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:48.133+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:48.426 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:48.422+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:48.513 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:48.510+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:48.524 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:48.521+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:48.971 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:48.967+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:49.160 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:49.156+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:49.177 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:49.175+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:49.398 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:49.394+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:49.505 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:49.502+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:49.541 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:49.539+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:49.543 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:49.541+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:49.951 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:49.947+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:50.200 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:50.198+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:50.215 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:50.212+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:50.357 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:50.353+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:50.481 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:50.478+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:50.511 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:50.508+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:50.513 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:50.510+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:50.961 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:50.957+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:51.182 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:51.179+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:51.205 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:51.202+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:51.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:51.331+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:51.474 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:51.468+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:51.475 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:51.471+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:51.513 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:51.510+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:51.954 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:51.950+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:52.162 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:52.159+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:52.186 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:52.183+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:52.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:52.331+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:52.523 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:52.519+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:52.524 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:52.521+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:52.539 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:52.535+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:52.975 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:52.971+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:53.181 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:53.179+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:53.185 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:53.183+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:53.343 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:53.339+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:53.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:53.506+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:53.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:53.507+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:53.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:53.587+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:53.980 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:53.977+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:54.174 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:54.171+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:54.220 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:54.217+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:54.374 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:54.371+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:54.491 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:54.488+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:54.494 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:54.491+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:54.609 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:54.606+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:55.018 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:55.014+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:55.191 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:55.188+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:55.271 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:55.268+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:55.340 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:55.336+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:55.478 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:55.474+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:55.478 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:55.476+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:55.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:55.627+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:56.051 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:56.049+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:56.202 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:56.199+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:56.231 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:56.229+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:56.369 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:56.367+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:56.496 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:56.493+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:56.497 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:56.496+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:56.653 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:56.650+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:57.027 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:57.024+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:57.236 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:57.233+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:57.240 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:57.237+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:57.330 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:57.327+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:57.486 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:57.480+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:57.488 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:57.482+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:57.666 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:57.664+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:58.022 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:58.019+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:58.199 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:58.196+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:58.210 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:58.207+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:58.296 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:58.293+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:58.466 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:58.463+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:58.468 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:58.466+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:58.674 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:58.672+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:58.975 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:58.974+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:42:59.185 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:42:59.183+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:42:59.259 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:42:59.257+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:42:59.270 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:42:59.268+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:42:59.506 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:59.502+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:42:59.506 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:42:59.504+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:42:59.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:42:59.725+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:42:59.937 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:42:59.934+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:00.195 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:00.192+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:00.237 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:00.234+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:00.248 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:00.246+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:00.529 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:00.525+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:00.529 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:00.527+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:00.742 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:00.739+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:00.930 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:00.928+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:01.208 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:01.205+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:01.239 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:01.236+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:01.242 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:01.239+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:01.548 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:01.546+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:01.551 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:01.549+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:01.765 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:01.763+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:01.952 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:01.950+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:02.226 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:02.222+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:02.235 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:02.233+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:02.284 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:02.281+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:02.603 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:02.599+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:02.604 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:02.602+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:02.726 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:02.724+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:02.932 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:02.928+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:03.203 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:03.200+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:03.233 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:03.231+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:03.257 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:03.253+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:03.653 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:03.651+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:03.654 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:03.652+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:03.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:03.743+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:03.945 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:03.942+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:04.210 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:04.206+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:04.212 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:04.210+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:04.307 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:04.305+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:04.630 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:04.627+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:04.630 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:04.628+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:04.800 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:04.796+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:04.985 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:04.981+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:05.236 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:05.234+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:05.240 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:05.236+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:05.339 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:05.337+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:05.611 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:05.608+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:05.612 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:05.609+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:05.799 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:05.796+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:05.997 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:05.993+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:06.207 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:06.204+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:06.277 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:06.274+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:06.384 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:06.382+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:06.631 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:06.628+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:06.632 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:06.630+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:06.770 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:06.766+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:06.965 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:06.961+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:07.238 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:07.234+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:07.279 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:07.276+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:07.345 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:07.342+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:07.624 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:07.621+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:07.627 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:07.625+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:07.793 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:07.789+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:07.957 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:07.950+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:08.240 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:08.237+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:08.284 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:08.281+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:08.305 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:08.302+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:08.607 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:08.604+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:08.610 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:08.607+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:08.758 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:08.755+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:08.935 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:08.926+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:09.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:09.208+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:09.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:09.274+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:09.307 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:09.302+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:09.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:09.596+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:09.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:09.597+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:09.782 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:09.778+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:09.898 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:09.895+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:10.187 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:10.184+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:10.249 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:10.247+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:10.331 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:10.328+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:10.581 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:10.577+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:10.581 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:10.579+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:10.791 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:10.787+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:10.866 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:10.862+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:11.175 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:11.171+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:11.219 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:11.217+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:11.331 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:11.328+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:11.552 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:11.543+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:11.558 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:11.546+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:11.810 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:11.806+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:11.856 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:11.851+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:12.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:12.155+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:12.263 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:12.261+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:12.311 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:12.307+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:12.534 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:12.530+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:12.534 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:12.532+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:12.855 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:12.852+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:12.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:12.900+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:13.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:13.154+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:13.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:13.290+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:13.344 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:13.341+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:13.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:13.569+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:13.574 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:13.571+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:13.903 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:13.900+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:13.906 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:13.902+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:14.142 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:14.139+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:14.314 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:14.310+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:14.370 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:14.367+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:14.587 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:14.583+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:14.587 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:14.585+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:14.867 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:14.863+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:14.876 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:14.873+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:15.152 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:43:15.149+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e572 v572) 2019-12-11T23:43:15.191 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:15.188+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:15.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:15.279+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:15.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:15.321+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:15.621 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:15.618+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:15.622 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:15.620+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:15.848 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:15.845+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:15.882 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:15.872+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:16.236 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:16.233+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:16.254 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:16.252+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:16.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:16.307+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:16.666 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:16.662+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:16.668 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:16.664+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:16.852 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:16.849+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:16.886 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:16.883+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:17.211 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:17.209+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:17.253 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:17.249+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:17.292 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:17.289+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:17.670 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:17.666+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:17.671 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:17.669+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:17.821 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:17.818+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:17.929 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:17.925+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:18.205 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:18.202+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:18.294 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:18.290+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:18.323 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:18.321+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:18.697 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:18.694+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:18.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:18.697+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:18.853 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:18.849+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:18.924 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:18.920+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:19.157 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:19.154+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:19.307 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:19.302+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:19.339 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:19.336+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:19.750 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:19.746+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:19.751 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:19.748+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:19.831 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:19.828+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:19.937 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:19.931+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:20.165 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:43:20.163+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e572 v572) 2019-12-11T23:43:20.172 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:20.170+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:20.268 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:20.266+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:20.296 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:20.293+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:20.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:20.788+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:20.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:20.791+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:20.872 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:20.869+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:20.917 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:20.914+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:21.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:21.211+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:21.252 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:21.248+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:21.341 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:21.338+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:21.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:21.751+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:21.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:21.754+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:21.889 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:21.886+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:21.964 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:21.960+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:22.175 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:22.173+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:22.291 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:22.287+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:22.368 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:22.364+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:22.748 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:22.745+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:22.751 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:22.749+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:22.858 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:22.854+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:22.932 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:22.928+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:23.168 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:23.166+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:23.286 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:23.283+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:23.404 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:23.401+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:23.764 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:23.760+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:23.765 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:23.763+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:23.886 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:23.883+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:23.984 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:23.980+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:24.192 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:24.190+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:24.335 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:24.332+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:24.425 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:43:24.423+0000 1694f700 -1 osd.4 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:54.881603+0000 front 2019-12-11T23:41:54.666308+0000 (oldest deadline 2019-12-11T23:42:40.535644+0000) 2019-12-11T23:43:24.784 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:24.780+0000 1694f700 -1 osd.0 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:42:08.990026+0000 (oldest deadline 2019-12-11T23:42:48.990026+0000) 2019-12-11T23:43:24.785 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:24.782+0000 1694f700 -1 osd.0 572 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:24.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:24.877+0000 1694f700 -1 osd.7 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.013329+0000 front 2019-12-11T23:41:56.233193+0000 (oldest deadline 2019-12-11T23:42:40.659537+0000) 2019-12-11T23:43:25.016 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:43:25.012+0000 1694f700 -1 osd.6 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.278215+0000 front 2019-12-11T23:41:56.265171+0000 (oldest deadline 2019-12-11T23:42:40.095909+0000) 2019-12-11T23:43:25.212 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:25.210+0000 1694f700 -1 osd.2 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:56.873895+0000 front 2019-12-11T23:41:56.868786+0000 (oldest deadline 2019-12-11T23:42:39.732980+0000) 2019-12-11T23:43:25.247 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:43:25.244+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e572 v572) 2019-12-11T23:43:25.305 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:25.298+0000 1694f700 -1 osd.1 572 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:41:57.583901+0000 front 2019-12-11T23:41:57.513376+0000 (oldest deadline 2019-12-11T23:42:38.611741+0000) 2019-12-11T23:43:26.148 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:43:26.062+0000 1694f700 -1 osd.0 573 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e569) 2019-12-11T23:43:51.659 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:51.656+0000 33f8b700 -1 osd.1 575 build_incremental_map_msg missing incremental map 564 2019-12-11T23:43:51.888 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:51.851+0000 3578e700 -1 osd.1 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.158 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:51.986+0000 31f87700 -1 osd.2 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.197 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:51.988+0000 31f87700 -1 osd.2 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.226 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:52.023+0000 36f91700 -1 osd.2 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.255 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:52.064+0000 32788700 -1 osd.2 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:52.288+0000 37391700 -1 osd.7 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.296 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:52.016+0000 37792700 -1 osd.1 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:52.321+0000 37391700 -1 osd.7 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.380 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:43:52.166+0000 32788700 -1 osd.2 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.384 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:52.087+0000 3578e700 -1 osd.1 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.582 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:52.389+0000 3638f700 -1 osd.7 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.596 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:52.327+0000 33f8b700 -1 osd.1 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.605 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:52.364+0000 37f93700 -1 osd.1 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.664 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:52.474+0000 37391700 -1 osd.7 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.676 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:52.526+0000 37792700 -1 osd.1 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:52.816 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:43:52.588+0000 37391700 -1 osd.7 576 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:58.357 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:43:58.352+0000 37f93700 -1 osd.1 578 build_incremental_map_msg missing incremental map 576 2019-12-11T23:43:58.532 INFO:teuthology.orchestra.run.smithi079.stderr:testBadHeaders (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:43:58.564 INFO:teuthology.orchestra.run.smithi079.stderr:testBlankMetadataName (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:44:12.255 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:11:59.188 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.261 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:11:59.192 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.276 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:11:59.208 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.282 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:11:59.186 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.285 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:11:59.218 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.303 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:11:59.208 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.555 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:11:59.488 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.571 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:11:59.504 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:12.707 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:11:59.612 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:44:54.835 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:44:54.829+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:44:54.858 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:44:54.852+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:44:55.026 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:44:55.023+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:44:55.822 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:44:55.815+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:44:55.848 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:44:55.843+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:44:55.853 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:44:55.849+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:44:56.050 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:44:56.047+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:44:56.866 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:44:56.861+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:44:56.880 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:44:56.876+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:44:56.894 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:44:56.891+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:44:57.089 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:44:57.086+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:44:57.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:44:57.795+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:44:57.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:44:57.873+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:44:57.887 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:44:57.882+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:44:57.899 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:44:57.897+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:44:58.023 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:44:58.020+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:44:58.088 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:44:58.085+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:44:58.804 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:44:58.799+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:44:58.857 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:44:58.853+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:44:58.919 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:44:58.916+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:44:58.932 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:44:58.926+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:44:59.048 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:44:59.044+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:44:59.104 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:44:59.095+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:44:59.850 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:44:59.847+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:44:59.889 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:44:59.885+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:44:59.909 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:44:59.904+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:44:59.918 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:44:59.912+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:00.096 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:00.093+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:00.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:00.136+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:00.812 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:00.809+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:00.855 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:00.852+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:00.918 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:00.914+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:00.949 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:00.946+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:01.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:01.136+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:01.166 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:01.162+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:01.825 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:01.822+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:01.830 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:01.826+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:01.936 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:01.932+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:01.968 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:01.965+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:02.131 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:02.127+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:02.143 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:02.140+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:02.846 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:02.842+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:02.866 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:02.863+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:02.942 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:02.939+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:02.974 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:02.971+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:03.144 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:03.140+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:03.145 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:03.140+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:03.843 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:03.839+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:03.895 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:03.893+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:03.957 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:03.954+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:04.012 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:04.009+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:04.139 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:04.135+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:04.160 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:04.157+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:04.882 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:04.878+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:04.946 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:04.943+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:04.973 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:04.969+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:04.991 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:04.987+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:05.154 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:05.150+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:05.197 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:05.187+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:05.860 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:05.858+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:05.929 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:05.926+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:05.935 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:05.931+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:05.970 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:05.967+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:06.164 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:06.159+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:06.189 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:06.185+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:06.894 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:06.886+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:06.958 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:06.954+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:06.973 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:06.969+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:07.022 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:07.016+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:07.152 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:07.148+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:07.240 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:07.235+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:07.927 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:07.923+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:07.982 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:07.980+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:07.983 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:07.979+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:07.992 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:07.990+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:08.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:08.111+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:08.267 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:08.263+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:08.893 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:08.890+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:08.978 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:08.974+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:08.980 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:08.977+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:09.025 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:09.021+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:09.166 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:09.163+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:09.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:09.276+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:09.872 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:09.869+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:10.005 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:10.003+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:10.008 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:10.006+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:10.029 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:10.027+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:10.158 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:10.155+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:10.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:10.241+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:10.894 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:10.891+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:11.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:11.013+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:11.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:11.035+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:11.071 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:11.068+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:11.143 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:11.140+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:11.243 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:11.239+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:11.879 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:11.876+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:12.000 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:11.997+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:12.027 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:12.024+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:12.077 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:12.072+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:12.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:12.139+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:12.291 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:12.280+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:12.925 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:12.922+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:12.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:12.969+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:13.055 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:13.052+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:13.065 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:13.062+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:13.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:13.192+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:13.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:13.320+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:13.902 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:13.899+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:13.981 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:13.978+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:14.014 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:14.010+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:14.073 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:14.071+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:14.176 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:14.173+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:14.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:14.295+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:14.871 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:14.868+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:15.012 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:15.010+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:15.027 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:15.024+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:15.044 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:15.040+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:15.169 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:15.165+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:15.341 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:15.337+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:15.846 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:15.842+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:15.971 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:15.968+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:16.062 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:16.058+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:16.083 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:16.073+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:16.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:16.177+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:16.354 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:16.349+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:16.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:16.798+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:16.975 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:16.972+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:17.074 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:17.070+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:17.131 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:17.126+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:17.166 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:17.163+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:17.364 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:17.361+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:17.780 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:17.778+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:17.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:17.987+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:18.081 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:18.079+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:18.106 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:18.103+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:18.153 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:18.148+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:18.413 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:18.401+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:18.759 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:18.756+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:19.024 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:19.021+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:19.108 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:19.103+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:19.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:19.118+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:19.146 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:19.143+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:19.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:19.432+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:19.759 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:19.750+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:19.998 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:19.995+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:20.112 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:20.108+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:20.123 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:20.119+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:20.186 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:20.182+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:20.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:20.425+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:20.746 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:20.744+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:21.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:21.012+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:21.131 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:21.128+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:21.151 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:21.148+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:21.224 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:21.220+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:21.432 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:21.429+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:21.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:21.794+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:21.979 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:21.977+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:22.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:22.084+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:22.171 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:22.168+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:22.255 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:22.251+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:22.436 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:22.432+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:22.808 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:22.805+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:22.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:22.930+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:23.133 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:23.129+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:23.149 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:23.145+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:23.237 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:23.233+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:23.403 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:23.399+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:23.853 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:23.849+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:23.918 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:23.915+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:24.147 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:24.143+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:24.190 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:24.187+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:24.200 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:24.197+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:24.396 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:24.392+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:24.860 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:24.858+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:24.950 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:24.947+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:25.138 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:25.134+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:25.176 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:25.173+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:25.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:25.191+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:25.404 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:25.400+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:25.607 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:45:25.605+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 42sec e578 v578) 2019-12-11T23:45:25.908 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:25.905+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:25.939 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:25.936+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:26.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:26.153+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:26.167 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:26.163+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:26.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:26.180+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:26.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:26.394+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:26.925 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:26.920+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:26.944 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:26.941+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:27.164 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:27.161+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:27.180 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:27.177+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:27.230 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:27.225+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:27.393 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:27.389+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:27.904 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:27.901+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:27.928 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:27.925+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:28.198 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:28.196+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:28.200 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:28.197+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:28.262 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:28.258+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:28.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:28.390+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:28.888 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:28.884+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:28.943 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:28.939+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:29.190 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:29.185+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:29.211 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:29.209+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:29.237 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:29.233+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:29.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:29.381+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:29.886 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:45:29.883+0000 1694f700 -1 osd.2 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.269944+0000 front 2019-12-11T23:44:12.192293+0000 (oldest deadline 2019-12-11T23:44:56.850774+0000) 2019-12-11T23:45:29.921 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:45:29.914+0000 1694f700 -1 osd.1 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.135712+0000 front 2019-12-11T23:44:12.270151+0000 (oldest deadline 2019-12-11T23:44:54.413513+0000) 2019-12-11T23:45:30.181 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:45:30.179+0000 1694f700 -1 osd.3 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:12.627434+0000 front 2019-12-11T23:44:12.686687+0000 (oldest deadline 2019-12-11T23:44:54.152120+0000) 2019-12-11T23:45:30.196 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:45:30.193+0000 1694f700 -1 osd.4 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:13.532817+0000 front 2019-12-11T23:44:13.752273+0000 (oldest deadline 2019-12-11T23:44:55.180656+0000) 2019-12-11T23:45:30.243 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:45:30.239+0000 1694f700 -1 osd.6 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:09.150469+0000 front 2019-12-11T23:44:09.109406+0000 (oldest deadline 2019-12-11T23:44:54.323490+0000) 2019-12-11T23:45:30.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:45:30.393+0000 1694f700 -1 osd.7 578 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:44:14.039576+0000 front 2019-12-11T23:44:14.009286+0000 (oldest deadline 2019-12-11T23:44:57.482821+0000) 2019-12-11T23:45:30.697 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:45:30.692+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 42sec e578 v578) 2019-12-11T23:46:29.070 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:29.067+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:29.379 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:29.375+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:30.123 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:30.118+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:30.394 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:30.389+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:30.696 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:30.692+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:30.854 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:30.850+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:31.127 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:31.123+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:31.361 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:31.358+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:31.704 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:31.700+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:31.900 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:31.889+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:31.973 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:31.970+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:32.178 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:32.174+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:32.419 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:32.412+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:32.520 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:32.518+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:32.663 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:32.660+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:32.913 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:32.909+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:32.992 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:32.989+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:33.200 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:33.197+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:33.420 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:33.417+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:33.475 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:33.473+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:33.676 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:33.673+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:33.964 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:33.961+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:33.969 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:33.965+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:34.193 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:34.189+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:34.449 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:34.446+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:34.449 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:34.443+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:34.655 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:34.652+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:34.919 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:34.916+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:34.988 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:34.985+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:35.225 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:35.222+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:35.427 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:35.424+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:35.488 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:35.486+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:35.642 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:35.638+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:35.949 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:35.942+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:35.958 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:35.956+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:36.225 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:36.219+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:36.441 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:36.436+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:36.497 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:36.492+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:36.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:36.628+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:36.941 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:36.939+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:36.987 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:36.985+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:37.225 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:37.212+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:37.477 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:37.474+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:37.499 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:37.497+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:37.628 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:37.624+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:37.956 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:37.939+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:37.965 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:37.950+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:38.258 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:38.254+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:38.434 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:38.430+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:38.482 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:38.477+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:38.483 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:38.481+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:38.592 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:38.588+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:38.922 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:38.919+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:38.940 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:38.937+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:39.302 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:39.297+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:39.437 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:39.435+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:39.438 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:39.437+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:39.456 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:39.453+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:39.553 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:39.550+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:39.953 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:39.950+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:39.973 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:39.971+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:40.336 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:40.333+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:40.415 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:40.413+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:40.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:40.454+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:40.459 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:40.456+0000 38b94700 -1 osd.7 584 build_incremental_map_msg missing incremental map 576 2019-12-11T23:46:40.459 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:40.456+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:40.509 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:40.506+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:40.945 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:40.942+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:40.969 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:40.967+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:41.321 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:41.318+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:41.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:41.413+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:41.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:41.414+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:41.423 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:41.419+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:41.532 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:41.527+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:41.933 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:41.931+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:41.947 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:41.944+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:42.312 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:42.308+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:42.399 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:42.396+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:42.419 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:42.417+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:42.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:42.419+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:42.516 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:42.509+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:42.906 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:42.903+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:42.953 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:42.951+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:43.359 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:43.356+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:43.360 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:43.357+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:43.444 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:43.441+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:43.447 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:43.445+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:43.528 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:43.524+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:43.945 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:43.942+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:43.951 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:43.949+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:44.348 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:44.342+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:44.348 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:44.345+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:44.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:44.406+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:44.411 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:44.408+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:44.542 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:44.539+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:44.939 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:44.937+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:44.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:44.953+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:45.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:45.345+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:45.402 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:45.398+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:45.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:45.433+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:45.439 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:45.436+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:45.512 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:45.508+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:45.910 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:45.907+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:45.921 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:45.918+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:46.371 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:46.367+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:46.381 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:46.378+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:46.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:46.411+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:46.416 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:46.414+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:46.555 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:46.551+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:46.933 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:46.930+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:46.956 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:46.947+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:47.385 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:47.381+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:47.391 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:47.388+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:47.391 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:47.389+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:47.402 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:47.399+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:47.608 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:47.606+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:47.914 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:47.912+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:47.958 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:47.954+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:48.345 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:48.342+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:48.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:48.369+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:48.375 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:48.371+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:48.453 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:48.451+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:48.639 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:48.637+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:48.913 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:48.910+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:48.933 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:48.927+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:49.373 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:49.356+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:49.385 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:49.370+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:49.386 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:49.371+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:49.437 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:49.430+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:49.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:49.666+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:49.926 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:49.923+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:49.980 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:49.977+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:50.392 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:50.390+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:50.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:50.397+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:50.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:50.399+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:50.455 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:50.454+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:50.649 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:50.645+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:50.889 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:50.884+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:51.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:51.014+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:51.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:51.365+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:51.368 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:51.367+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:51.386 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:51.382+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:51.503 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:51.502+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:51.680 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:51.678+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:51.938 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:51.935+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:52.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:52.046+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:52.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:52.328+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:52.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:52.329+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:52.400 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:52.398+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:52.499 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:52.495+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:52.704 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:52.702+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:52.953 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:52.951+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:53.015 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:53.013+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:53.311 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:53.306+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:53.312 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:53.309+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:53.387 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:53.385+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:53.545 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:53.543+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:53.739 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:53.737+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:53.982 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:53.980+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:53.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:53.981+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:54.357 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:54.355+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:54.360 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:54.358+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:54.404 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:54.402+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:54.579 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:54.576+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:54.750 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:54.748+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:54.969 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:54.966+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:54.983 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:54.980+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:55.319 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:55.316+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:55.320 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:55.317+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:55.393 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:55.390+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:55.556 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:55.552+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:55.725 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:55.722+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:55.965 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:55.963+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:55.968 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:55.963+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:56.287 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:56.284+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:56.289 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:56.288+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:56.406 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:56.402+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:56.587 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:56.585+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:56.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:56.743+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:56.949 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:56.947+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:57.005 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:57.002+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:57.299 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:57.296+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:57.300 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:57.296+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:57.433 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:57.430+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:57.563 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:57.560+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:57.778 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:57.775+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:57.988 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:57.985+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:58.006 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:58.003+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:58.337 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:58.335+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:58.339 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:58.337+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:58.426 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:58.422+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:58.544 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:58.540+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:58.736 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:58.733+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:46:59.021 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:46:59.018+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:46:59.037 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:46:59.033+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:46:59.316 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:59.309+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:46:59.316 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:46:59.311+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:46:59.460 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:46:59.456+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:46:59.519 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:46:59.516+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:46:59.769 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:46:59.766+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:00.005 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:00.000+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:00.040 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:00.031+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:00.348 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:00.336+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:00.348 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:00.337+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:00.482 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:00.476+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:00.514 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:00.509+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:00.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:00.805+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:00.938 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:47:00.932+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e584 v584) 2019-12-11T23:47:01.030 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:01.028+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:01.080 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:01.075+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:01.313 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:01.311+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:01.315 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:01.313+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:01.449 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:01.446+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:01.519 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:01.516+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:01.812 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:01.809+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:02.001 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:01.998+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:02.048 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:02.046+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:02.275 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:02.272+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:02.275 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:02.273+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:02.413 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:02.410+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:02.558 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:02.555+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:02.780 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:02.776+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:03.032 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:03.030+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:03.043 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:03.041+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:03.248 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:03.245+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:03.249 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:03.246+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:03.404 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:03.401+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:03.584 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:03.581+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:03.762 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:03.759+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:04.068 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:04.066+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:04.071 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:04.068+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:04.277 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:04.274+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:04.278 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:04.275+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:04.365 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:04.363+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:04.612 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:04.608+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:04.789 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:04.786+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:05.030 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:05.027+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:05.106 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:05.103+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:05.280 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:05.276+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:05.280 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:05.277+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:05.357 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:05.354+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:05.613 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:05.609+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:05.808 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:05.804+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:05.947 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:47:05.943+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e584 v584) 2019-12-11T23:47:06.034 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:06.030+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:06.134 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:06.131+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:06.233 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:06.230+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:06.233 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:06.231+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:06.383 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:06.379+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:06.616 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:06.613+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:06.839 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:06.836+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:07.009 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:07.006+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:07.088 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:07.085+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:07.267 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:07.264+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:07.268 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:07.265+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:07.372 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:07.368+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:07.585 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:07.582+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:07.883 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:07.879+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:08.023 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:08.018+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:08.085 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:08.081+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:08.232 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:08.228+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:08.236 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:08.232+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:08.362 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:08.358+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:08.609 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:08.607+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:08.929 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:08.925+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:09.049 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:09.044+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:09.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:09.045+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:09.195 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:09.191+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:09.195 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:09.192+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:09.372 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:09.368+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:09.607 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:09.604+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:09.967 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:09.963+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:10.032 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:10.029+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:10.072 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:10.068+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:10.170 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:10.166+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:10.171 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:10.168+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:10.361 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:10.357+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:10.652 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:10.648+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:10.957 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:47:10.955+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e584 v584) 2019-12-11T23:47:11.009 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:11.006+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:11.019 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:11.015+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:11.066 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:11.063+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:11.213 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:11.209+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:11.213 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:11.210+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:11.316 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:11.313+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:11.645 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:11.642+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:12.012 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:12.009+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:12.024 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:12.021+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:12.025 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:12.022+0000 1694f700 -1 osd.7 584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+write+known_if_redirected e584) 2019-12-11T23:47:12.066 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:12.062+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:12.232 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:12.228+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:12.233 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:12.231+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:12.326 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:12.323+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:12.695 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:12.692+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:13.025 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:13.021+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:13.027 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:13.025+0000 1694f700 -1 osd.7 584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+write+known_if_redirected e584) 2019-12-11T23:47:13.029 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:13.025+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:13.038 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:13.036+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:13.237 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:13.232+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:13.237 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:13.235+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:13.363 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:13.359+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:13.687 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:13.684+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:14.000 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:13.996+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:14.001 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:13.998+0000 1694f700 -1 osd.7 584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+write+known_if_redirected e584) 2019-12-11T23:47:14.049 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:14.046+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:14.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:14.063+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:14.217 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:14.214+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:14.217 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:14.215+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:14.381 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:14.377+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:14.645 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:14.642+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:15.003 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:15.002+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:15.004 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:15.001+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:15.007 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:15.005+0000 1694f700 -1 osd.7 584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+write+known_if_redirected e584) 2019-12-11T23:47:15.052 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:15.049+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:15.209 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:15.207+0000 1694f700 -1 osd.0 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:45:58.216969+0000 (oldest deadline 2019-12-11T23:46:38.216969+0000) 2019-12-11T23:47:15.211 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:15.210+0000 1694f700 -1 osd.0 584 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:47:15.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:47:15.403+0000 1694f700 -1 osd.4 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:43.102292+0000 front 2019-12-11T23:45:43.106282+0000 (oldest deadline 2019-12-11T23:46:28.749658+0000) 2019-12-11T23:47:15.649 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:47:15.646+0000 1694f700 -1 osd.6 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:44.790849+0000 front 2019-12-11T23:45:44.706053+0000 (oldest deadline 2019-12-11T23:46:28.179469+0000) 2019-12-11T23:47:15.968 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:15.964+0000 1694f700 -1 osd.7 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.105966+0000 front 2019-12-11T23:45:47.179484+0000 (oldest deadline 2019-12-11T23:46:29.994911+0000) 2019-12-11T23:47:15.969 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:47:15.966+0000 1694f700 -1 osd.7 584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+write+known_if_redirected e584) 2019-12-11T23:47:15.997 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:47:15.994+0000 1694f700 -1 osd.1 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:47.704303+0000 front 2019-12-11T23:45:47.776619+0000 (oldest deadline 2019-12-11T23:46:30.565093+0000) 2019-12-11T23:47:16.047 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:47:16.033+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e584 v584) 2019-12-11T23:47:16.100 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:47:16.097+0000 1694f700 -1 osd.2 584 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:45:46.774410+0000 front 2019-12-11T23:45:46.785320+0000 (oldest deadline 2019-12-11T23:46:31.448607+0000) 2019-12-11T23:47:17.066 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:47:17.056+0000 1694f700 -1 osd.0 585 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e582) 2019-12-11T23:48:06.484 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:15:53.417 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.488 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:15:53.421 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.497 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:15:53.430 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.509 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:15:53.436 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.511 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:15:53.416 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.523 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:15:53.428 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.758 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:15:53.691 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.834 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:15:53.766 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:06.858 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:15:53.763 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:14.200 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:14.196+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:15.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:15.179+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:16.184 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:16.182+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:17.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:17.193+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:18.247 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:18.245+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:19.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:19.292+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:20.317 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:20.315+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:21.328 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:21.326+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:22.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:22.341+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:23.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:23.330+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:24.384 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:24.381+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:25.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:25.388+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:26.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:26.441+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:27.422 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:27.419+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:28.395 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:28.393+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:29.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:29.370+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:30.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:30.349+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:31.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:31.327+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:31.470 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:16:18.404 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.514 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:16:18.447 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.545 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:16:18.450 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.550 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:16:18.455 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.565 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:16:18.497 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.567 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:16:18.500 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.568 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:16:18.501 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.596 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:16:18.528 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:31.620 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:16:18.524 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:32.301 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:32.298+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:33.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:33.323+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:34.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:34.348+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:35.355 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:35.352+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:36.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:36.308+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:37.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:37.277+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:38.238 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:38.236+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:39.236 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:39.234+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:40.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:40.232+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:41.263 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:41.261+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:42.257 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:42.256+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:43.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:43.294+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:44.312 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:44.311+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:44.935 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:44.933+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:45.333 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:45.329+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:45.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:45.336+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:45.346 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:45.345+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:45.933 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:45.930+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:46.069 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:46.066+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:46.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:46.323+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:46.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:46.326+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:46.378 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:46.375+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:46.398 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:46.394+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:46.979 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:46.976+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:47.044 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:47.041+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:47.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:47.346+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:47.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:47.351+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:47.398 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:47.389+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:47.409 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:47.405+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:47.990 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:47.988+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:48.101 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:48.094+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:48.179 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:48.176+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:48.367 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:48.363+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:48.377 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:48.373+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:48.377 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:48.376+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:48.432 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:48.430+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:49.018 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:49.012+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:49.092 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:49.090+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:49.202 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:49.199+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:49.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:49.394+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:49.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:49.397+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:49.419 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:49.415+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:49.447 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:49.442+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:50.014 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:50.010+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:50.124 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:50.120+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:50.237 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:50.233+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:50.403 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:50.397+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:50.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:50.405+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:50.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:50.407+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:50.468 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:50.466+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:51.029 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:51.025+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:51.111 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:51.108+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:51.227 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:51.224+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:51.447 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:51.441+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:51.448 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:51.443+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:51.449 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:51.445+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:51.468 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:51.465+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:52.025 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:52.021+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:52.127 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:52.124+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:52.247 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:52.244+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:52.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:52.453+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:52.473 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:52.469+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:52.474 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:52.471+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:52.500 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:52.492+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:53.067 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:53.063+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:53.181 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:53.178+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:53.222 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:53.219+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:53.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:53.441+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:53.446 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:53.445+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:53.491 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:53.484+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:53.553 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:53.543+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:54.028 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:54.025+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:54.187 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:54.184+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:54.225 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:54.222+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:54.449 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:54.446+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:54.478 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:54.474+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:54.480 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:54.479+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:54.505 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:54.501+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:54.987 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:54.983+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:55.190 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:55.187+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:55.212 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:55.209+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:55.443 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:55.441+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:55.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:55.482+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:55.487 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:55.485+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:55.524 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:55.520+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:55.955 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:55.951+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:56.208 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:56.195+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:56.208 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:56.198+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:56.431 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:56.427+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:56.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:56.497+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:56.504 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:56.502+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:56.520 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:56.516+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:56.828 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:16:43.761 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.833 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:16:43.766 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.875 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:16:43.808 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.877 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:16:43.810 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.888 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:16:43.821 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.927 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:16:43.832 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.931 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:16:43.836 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.932 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:16:43.865 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:56.935 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:56.933+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:57.032 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:16:43.937 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:48:57.230 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:57.227+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:57.233 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:57.229+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:57.430 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:57.427+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:57.478 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:57.476+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:57.523 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:57.519+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:57.524 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:57.521+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:57.916 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:57.912+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:58.194 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:58.192+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:58.270 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:58.267+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:58.382 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:58.379+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:58.468 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:58.464+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:58.496 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:58.493+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:58.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:58.497+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:58.925 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:58.922+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:48:59.208 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:48:59.205+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:48:59.248 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:48:59.242+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:48:59.382 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:48:59.378+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:48:59.421 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:48:59.418+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:48:59.493 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:59.485+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:48:59.494 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:48:59.487+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:48:59.956 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:48:59.951+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:00.179 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:00.175+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:00.251 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:00.245+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:00.399 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:00.395+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:00.425 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:00.423+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:00.528 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:00.524+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:00.529 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:00.527+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:00.957 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:00.954+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:01.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:01.210+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:01.214 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:01.211+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:01.443 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:01.440+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:01.477 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:01.475+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:01.509 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:01.493+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:01.509 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:01.498+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:01.955 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:01.952+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:02.171 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:02.168+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:02.200 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:02.198+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:02.441 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:02.437+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:02.505 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:02.499+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:02.510 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:02.507+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:02.517 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:02.512+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:03.007 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:03.003+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:03.146 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:03.141+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:03.190 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:03.186+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:03.406 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:03.402+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:03.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:03.497+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:03.504 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:03.502+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:03.527 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:03.524+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:04.034 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:04.031+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:04.107 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:04.104+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:04.188 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:04.185+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:04.367 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:04.363+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:04.461 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:04.455+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:04.461 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:04.458+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:04.525 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:04.520+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:05.083 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:05.079+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:05.087 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:05.084+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:05.206 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:05.203+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:05.375 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:05.371+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:05.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:05.483+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:05.489 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:05.488+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:05.515 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:05.512+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:06.097 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:06.095+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:06.098 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:06.094+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:06.201 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:06.198+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:06.421 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:06.417+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:06.510 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:06.507+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:06.513 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:06.511+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:06.550 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:06.547+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:07.066 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:07.063+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:07.066 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:07.064+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:07.220 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:07.217+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:07.387 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:07.384+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:07.519 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:07.515+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:07.526 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:07.521+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:07.530 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:07.524+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:08.035 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:08.031+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:08.101 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:08.099+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:08.245 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:08.242+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:08.410 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:08.406+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:08.482 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:08.479+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:08.491 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:08.484+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:08.512 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:08.508+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:09.083 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:09.079+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:09.125 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:09.122+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:09.269 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:09.266+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:09.384 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:09.378+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:09.482 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:09.478+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:09.505 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:09.501+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:09.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:09.505+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:10.076 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:10.062+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:10.107 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:10.094+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:10.324 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:10.313+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:10.354 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:10.344+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:10.507 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:10.504+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:10.518 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:10.513+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:10.519 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:10.516+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:11.069 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:11.066+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:11.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:11.141+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:11.325 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:11.321+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:11.372 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:11.368+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:11.507 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:11.503+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:11.510 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:11.508+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:11.560 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:11.556+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:12.031 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:12.027+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:12.100 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:12.098+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:12.373 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:12.369+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:12.381 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:12.376+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:12.508 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:12.504+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:12.509 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:12.506+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:12.525 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:12.521+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:12.987 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:12.983+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:13.084 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:13.081+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:13.375 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:13.368+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:13.409 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:13.405+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:13.513 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:13.507+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:13.513 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:13.509+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:13.572 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:13.568+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:14.028 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:14.024+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:14.049 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:14.045+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:14.419 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:14.413+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:14.434 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:14.432+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:14.558 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:14.554+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:14.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:14.557+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:14.601 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:14.596+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:15.005 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:15.002+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:15.079 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:15.075+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:15.413 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:15.410+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:15.436 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:15.434+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:15.536 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:15.532+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:15.537 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:15.535+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:15.576 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:15.571+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:15.998 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:15.995+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:16.043 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:16.039+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:16.424 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:16.420+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:16.424 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:16.421+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:16.551 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:16.548+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:16.551 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:16.550+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:16.561 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:16.558+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:16.818 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:49:16.812+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 46sec e590 v590) 2019-12-11T23:49:17.007 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:17.003+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:17.046 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:17.041+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:17.425 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:17.421+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:17.441 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:17.437+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:17.553 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:17.548+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:17.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:17.551+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:17.574 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:17.570+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:18.028 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:18.025+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:18.058 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:18.054+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:18.409 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:18.403+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:18.410 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:18.405+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:18.530 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:18.526+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:18.546 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:18.543+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:18.550 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:18.548+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:19.015 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:19.011+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:19.106 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:19.102+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:19.374 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:19.370+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:19.414 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:19.410+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:19.525 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:19.521+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:19.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:19.554+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:19.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:19.557+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:20.019 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:20.016+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:20.118 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:20.115+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:20.414 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:20.410+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:20.421 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:20.418+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:20.518 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:20.513+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:20.548 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:20.541+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:20.549 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:20.543+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:21.020 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:49:21.014+0000 1694f700 -1 osd.6 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.312010+0000 front 2019-12-11T23:48:02.097351+0000 (oldest deadline 2019-12-11T23:48:44.912615+0000) 2019-12-11T23:49:21.160 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:49:21.156+0000 1694f700 -1 osd.2 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.993665+0000 front 2019-12-11T23:48:02.932209+0000 (oldest deadline 2019-12-11T23:48:48.160214+0000) 2019-12-11T23:49:21.380 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:49:21.377+0000 1694f700 -1 osd.3 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:01.383867+0000 front 2019-12-11T23:48:01.386102+0000 (oldest deadline 2019-12-11T23:48:45.440680+0000) 2019-12-11T23:49:21.432 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:49:21.429+0000 1694f700 -1 osd.4 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.322466+0000 front 2019-12-11T23:47:59.321396+0000 (oldest deadline 2019-12-11T23:48:44.350753+0000) 2019-12-11T23:49:21.497 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:21.493+0000 1694f700 -1 osd.1 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:48:02.073051+0000 front 2019-12-11T23:48:02.081531+0000 (oldest deadline 2019-12-11T23:48:45.383150+0000) 2019-12-11T23:49:21.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:21.496+0000 1694f700 -1 osd.7 590 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:47:59.330016+0000 front 2019-12-11T23:47:59.341274+0000 (oldest deadline 2019-12-11T23:48:44.415809+0000) 2019-12-11T23:49:21.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:21.499+0000 1694f700 -1 osd.7 590 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:21.881 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:49:21.863+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 46sec e590 v590) 2019-12-11T23:49:22.646 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:49:22.563+0000 1694f700 -1 osd.7 591 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e588) 2019-12-11T23:49:53.185 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:53.089+0000 34f8d700 -1 osd.1 595 build_incremental_map_msg missing incremental map 595 2019-12-11T23:49:53.235 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:53.123+0000 3578e700 -1 osd.1 595 build_incremental_map_msg missing incremental map 595 2019-12-11T23:49:53.259 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:53.188+0000 34f8d700 -1 osd.1 595 build_incremental_map_msg missing incremental map 595 2019-12-11T23:49:53.615 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:53.430+0000 35f8f700 -1 osd.1 595 build_incremental_map_msg missing incremental map 595 2019-12-11T23:49:53.670 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:49:53.451+0000 36790700 -1 osd.1 595 build_incremental_map_msg missing incremental map 595 2019-12-11T23:50:20.576 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:20.572+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:20.847 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:20.843+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:21.539 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:21.536+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:21.851 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:21.847+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:22.045 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:22.041+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:22.505 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:22.501+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:22.851 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:22.847+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:23.017 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:23.013+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:23.037 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:23.034+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:23.398 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:23.394+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:23.531 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:23.528+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:23.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:23.901+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:24.043 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:24.039+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:24.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:24.041+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:24.075 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:24.071+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:24.439 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:24.437+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:24.488 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:24.485+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:24.953 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:24.950+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:25.015 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:25.012+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:25.016 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:25.013+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:25.062 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:25.060+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:25.341 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:25.339+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:25.415 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:25.413+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:25.498 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:25.495+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:25.938 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:25.935+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:25.996 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:25.992+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:25.996 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:25.993+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:26.018 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:26.015+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:26.393 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:26.391+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:26.411 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:26.409+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:26.537 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:26.533+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:26.946 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:26.940+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:26.962 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:26.958+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:26.962 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:26.960+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:26.978 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:26.975+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:27.378 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:27.376+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:27.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:27.382+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:27.566 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:27.563+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:27.965 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:27.961+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:27.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:27.967+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:27.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:27.972+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:27.982 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:27.976+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:28.335 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:28.333+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:28.353 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:28.350+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:28.616 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:28.613+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:28.924 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:28.921+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:28.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:28.931+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:28.936 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:28.934+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:28.949 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:28.946+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:29.326 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:29.324+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:29.351 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:29.349+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:29.636 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:29.633+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:29.934 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:29.932+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:29.950 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:29.947+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:29.957 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:29.952+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:29.972 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:29.967+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:30.279 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:30.277+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:30.351 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:30.349+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:30.615 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:30.612+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:30.917 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:30.913+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:30.923 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:30.915+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:30.960 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:30.956+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:30.978 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:30.975+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:31.256 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:31.255+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:31.258 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:31.257+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:31.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:31.323+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:31.645 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:31.642+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:31.944 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:31.940+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:31.944 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:31.942+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:31.992 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:31.988+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:32.008 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:32.005+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:32.236 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:32.233+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:32.239 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:32.237+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:32.284 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:32.281+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:32.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:32.683+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:32.966 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:32.963+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:32.966 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:32.963+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:32.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:32.989+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:32.995 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:32.993+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:33.218 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:33.215+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:33.219 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:33.216+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:33.256 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:33.254+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:33.650 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:33.647+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:33.990 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:33.987+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:33.993 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:33.990+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:34.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:34.015+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:34.019 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:34.016+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:34.209 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:34.204+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:34.209 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:34.206+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:34.233 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:34.231+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:34.687 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:34.679+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:35.004 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:35.000+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:35.004 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:35.001+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:35.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:35.002+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:35.042 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:35.039+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:35.180 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:35.177+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:35.182 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:35.179+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:35.198 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:35.195+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:35.696 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:35.693+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:35.995 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:35.993+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:35.996 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:35.995+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:36.012 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:36.010+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:36.054 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:36.051+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:36.146 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:36.143+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:36.146 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:36.144+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:36.225 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:36.222+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:36.659 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:36.656+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:37.010 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:37.008+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:37.011 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:37.008+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:37.011 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:37.009+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:37.021 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:37.019+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:37.189 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:37.186+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:37.190 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:37.188+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:37.244 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:37.242+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:37.672 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:37.670+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:37.972 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:37.969+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:37.982 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:37.980+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:37.983 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:37.982+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:38.023 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:38.021+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:38.240 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:38.238+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:38.241 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:38.239+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:38.268 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:38.265+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:38.684 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:38.682+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:39.019 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:39.016+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:39.032 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:39.029+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:39.037 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:39.034+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:39.037 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:39.036+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:39.211 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:39.209+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:39.211 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:39.209+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:39.261 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:39.258+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:39.669 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:39.666+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:39.980 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:39.977+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:40.068 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:40.066+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:40.081 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:40.079+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:40.082 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:40.081+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:40.254 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:40.252+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:40.256 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:40.255+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:40.276 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:40.275+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:40.669 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:40.666+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:40.967 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:40.964+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:41.099 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:41.097+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:41.113 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:41.112+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:41.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:41.115+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:41.232 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:41.229+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:41.232 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:41.230+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:41.319 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:41.317+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:41.677 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:41.675+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:41.968 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:41.965+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:42.086 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:42.084+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:42.125 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:42.123+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:42.128 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:42.127+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:42.201 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:42.199+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:42.201 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:42.200+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:42.274 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:42.272+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:42.643 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:42.641+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:42.937 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:42.935+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:43.069 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:43.067+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:43.100 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:43.098+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:43.101 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:43.100+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:43.167 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:43.165+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:43.168 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:43.166+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:43.260 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:43.258+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:43.636 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:43.632+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:43.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:43.952+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:44.059 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:44.057+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:44.059 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:44.058+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:44.069 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:44.066+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:44.157 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:44.152+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:44.158 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:44.153+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:44.303 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:44.288+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:44.687 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:44.684+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:44.962 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:44.957+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:45.047 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:45.043+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:45.048 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:45.045+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:45.121 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:45.117+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:45.139 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:45.134+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:45.139 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:45.136+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:45.258 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:45.255+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:45.710 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:45.707+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:46.013 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:46.008+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:46.055 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:46.051+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:46.055 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:46.053+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:46.092 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:46.085+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:46.186 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:46.183+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:46.189 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:46.187+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:46.265 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:46.262+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:46.693 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:46.691+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:47.028 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:47.023+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:47.033 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:47.028+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:47.038 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:47.030+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:47.085 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:47.082+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:47.183 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:47.180+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:47.184 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:47.182+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:47.274 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:47.270+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:47.652 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:47.648+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:48.030 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:48.026+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:48.031 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:48.028+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:48.031 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:48.027+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:48.100 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:48.098+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:48.168 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:48.163+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:48.169 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:48.166+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:48.296 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:48.293+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:48.693 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:48.690+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:49.015 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:49.011+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:49.016 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:49.012+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:49.016 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:49.013+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:49.097 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:49.094+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:49.158 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:49.154+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:49.158 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:49.155+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:49.279 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:49.276+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:49.647 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:49.645+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:49.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:49.982+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:50.064 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:50.060+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:50.065 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:50.062+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:50.117 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:50.113+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:50.126 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:50.121+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:50.135 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:50.132+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:50.321 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:50.318+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:50.636 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:50.633+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:50.989 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:50.987+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:51.057 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:51.054+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:51.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:51.056+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:51.115 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:51.112+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:51.119 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:51.115+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:51.160 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:51.157+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:51.326 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:51.323+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:51.688 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:51.669+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:51.966 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:51.962+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:52.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:52.103+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:52.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:52.105+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:52.142 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:52.136+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:52.146 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:52.141+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:52.190 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:52.187+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:52.349 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:52.347+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:52.676 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:52.673+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:53.015 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:53.012+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:53.126 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:53.123+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:53.129 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:53.127+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:53.153 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:53.150+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:53.180 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:53.177+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:53.181 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:53.179+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:53.322 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:53.318+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:53.714 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:53.711+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:54.067 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:54.065+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:54.113 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:54.109+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:54.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:54.173+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:54.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:54.176+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:54.196 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:54.193+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:54.196 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:54.194+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:54.333 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:54.331+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:54.719 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:54.716+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:55.047 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:55.044+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:55.132 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:55.128+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:55.182 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:55.179+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:55.183 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:55.181+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:55.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:55.198+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:55.204 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:55.202+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:55.297 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:55.292+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:55.745 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:55.740+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:56.049 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:56.047+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:56.137 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:56.135+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:56.137 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:56.136+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:56.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:56.179+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:56.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:56.230+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:56.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:56.232+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:56.338 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:56.336+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:56.716 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:56.713+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:57.099 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:57.097+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:57.128 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:57.126+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:57.130 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:57.128+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:57.136 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:50:57.133+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e597 v597) 2019-12-11T23:50:57.150 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:57.147+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:57.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:57.246+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:57.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:57.248+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:57.299 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:57.297+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:57.763 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:57.762+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:58.082 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:58.080+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:58.095 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:58.092+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:58.097 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:58.095+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:58.121 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:58.117+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:58.231 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:58.227+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:58.231 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:58.229+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:58.345 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:58.342+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:58.753 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:58.750+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:50:59.053 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:59.051+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:50:59.053 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:50:59.051+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:50:59.083 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:50:59.079+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:50:59.121 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:50:59.119+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:50:59.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:59.268+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:50:59.272 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:50:59.269+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:50:59.382 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:50:59.379+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:50:59.743 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:50:59.739+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:00.026 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:00.022+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:00.027 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:00.026+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:00.076 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:00.073+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:00.141 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:00.096+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:00.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:00.262+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:51:00.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:00.266+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:00.351 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:51:00.348+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:51:00.772 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:00.767+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:01.067 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:01.064+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:01.073 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:01.070+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:01.074 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:01.072+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:01.101 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:01.098+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:01.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:01.286+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:51:01.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:01.288+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:01.373 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:51:01.370+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:51:01.759 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:01.756+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:02.062 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:02.057+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:02.094 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:02.091+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:02.119 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:02.116+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:02.121 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:02.119+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:02.146 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:51:02.143+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e597 v597) 2019-12-11T23:51:02.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:02.279+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:51:02.285 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:02.281+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:02.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:51:02.323+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:51:02.735 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:02.733+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:03.074 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:03.071+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:03.075 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:03.073+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:03.098 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:03.094+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:03.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:03.102+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:03.289 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:51:03.286+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:51:03.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:03.325+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:51:03.329 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:03.327+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:03.710 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:03.707+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:04.053 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:04.051+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:04.079 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:04.077+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:04.105 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:04.101+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:04.105 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:04.103+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:04.321 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:51:04.317+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:51:04.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:04.331+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:51:04.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:04.333+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:04.733 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:04.730+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:05.053 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:05.050+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:05.075 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:05.073+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:05.077 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:05.075+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:05.079 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:05.077+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:05.292 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:05.289+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:51:05.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:05.293+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:05.323 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:51:05.321+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:51:05.736 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:05.733+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:06.080 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:06.077+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:06.082 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:06.080+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:06.097 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:06.092+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:06.097 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:06.093+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:06.328 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:06.325+0000 1694f700 -1 osd.7 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.982615+0000 front 2019-12-11T23:49:38.837261+0000 (oldest deadline 2019-12-11T23:50:22.882315+0000) 2019-12-11T23:51:06.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:06.327+0000 1694f700 -1 osd.7 597 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:06.339 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:51:06.336+0000 1694f700 -1 osd.1 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:38.732223+0000 front 2019-12-11T23:49:39.003572+0000 (oldest deadline 2019-12-11T23:50:22.741153+0000) 2019-12-11T23:51:06.758 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:06.755+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:07.043 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:51:07.041+0000 1694f700 -1 osd.6 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.395303+0000 front 2019-12-11T23:49:36.116809+0000 (oldest deadline 2019-12-11T23:50:20.197053+0000) 2019-12-11T23:51:07.058 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:07.054+0000 1694f700 -1 osd.0 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:49:50.727763+0000 (oldest deadline 2019-12-11T23:50:30.727763+0000) 2019-12-11T23:51:07.058 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:51:07.056+0000 1694f700 -1 osd.0 597 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e595) 2019-12-11T23:51:07.102 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:51:07.099+0000 1694f700 -1 osd.2 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:36.245802+0000 front 2019-12-11T23:49:36.281755+0000 (oldest deadline 2019-12-11T23:50:21.534189+0000) 2019-12-11T23:51:07.250 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:51:07.224+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e597 v597) 2019-12-11T23:51:07.377 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:51:07.369+0000 1694f700 -1 osd.7 598 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:119318 2.2fs0 2.80d35af (undecoded) ondisk+retry+write+known_if_redirected e595) 2019-12-11T23:51:07.726 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:51:07.723+0000 1694f700 -1 osd.4 597 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:49:35.763974+0000 front 2019-12-11T23:49:35.618517+0000 (oldest deadline 2019-12-11T23:50:20.249006+0000) 2019-12-11T23:51:45.111 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:19:32.044 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.116 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:19:32.050 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.125 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:19:32.058 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.128 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:19:32.061 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.138 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:19:32.042 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.158 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:19:32.055 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.328 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:19:32.262 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.393 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:19:32.325 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:51:45.460 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:19:32.365 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:52:00.350 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:00.346+0000 37f93700 -1 osd.1 603 build_incremental_map_msg missing incremental map 595 2019-12-11T23:52:01.121 INFO:teuthology.orchestra.run.smithi079.stderr:testChunkedPut (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:52:08.375 INFO:teuthology.orchestra.run.smithi079.stderr:testContentTypeGuessing (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:52:14.779 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:14.763+0000 3478c700 -1 osd.2 603 build_incremental_map_msg missing incremental map 576 2019-12-11T23:52:33.934 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:33.930+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:33.960 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:33.956+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:34.194 INFO:teuthology.orchestra.run.smithi079.stderr:testCopy (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:52:34.411 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:34.408+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:34.941 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:34.939+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:34.944 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:34.941+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:35.391 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:35.388+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:35.690 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:35.687+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:35.963 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:35.958+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:35.978 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:35.975+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:36.211 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:36.209+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:36.429 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:36.427+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:36.755 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:36.752+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:36.997 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:36.993+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:37.000 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:36.996+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:37.189 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:37.185+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:37.394 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:37.391+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:37.722 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:37.719+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:37.745 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:37.743+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:37.969 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:37.965+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:37.988 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:37.986+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:38.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:38.144+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:38.389 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:38.382+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:38.712 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:38.708+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:38.757 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:38.754+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:39.004 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:39.000+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:39.018 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:39.016+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:39.131 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:39.128+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:39.419 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:39.415+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:39.675 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:39.673+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:39.799 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:39.796+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:40.042 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:40.039+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:40.062 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:40.058+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:40.136 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:40.132+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:40.465 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:40.461+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:40.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:40.651+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:40.795 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:40.792+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:41.087 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:41.082+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:41.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:41.081+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:41.123 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:41.118+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:41.465 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:41.461+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:41.668 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:41.655+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:41.747 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:41.744+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:42.083 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:42.079+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:42.090 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:42.084+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:42.117 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:42.113+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:42.432 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:42.428+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:42.621 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:42.619+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:42.771 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:42.768+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:43.043 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:43.040+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:43.110 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:43.106+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:43.135 INFO:teuthology.orchestra.run.smithi079.stderr:testCopy404s (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:52:43.146 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:43.143+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:43.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:43.403+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:43.664 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:43.660+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:43.803 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:43.799+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:44.077 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:44.074+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:44.123 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:44.120+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:44.133 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:44.129+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:44.423 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:44.419+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:44.717 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:44.714+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:44.794 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:44.791+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:45.113 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:45.111+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:45.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:45.112+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:45.177 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:45.173+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:45.460 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:45.456+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:45.605 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyDestinationSlashProblems (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:52:45.725 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:45.721+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:45.797 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:45.795+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:46.138 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:46.130+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:46.162 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:46.158+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:46.169 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:46.164+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:46.441 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:46.434+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:46.737 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:46.730+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:46.811 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:46.807+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:47.121 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:47.115+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:47.155 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:47.151+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:47.161 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:47.158+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:47.395 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:47.392+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:47.771 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:47.767+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:47.852 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:47.848+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:48.075 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:48.071+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:48.117 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:48.113+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:48.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:48.178+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:48.412 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:48.408+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:48.779 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:48.777+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:48.856 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:48.852+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:49.040 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:49.038+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:49.109 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:49.105+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:49.192 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:49.188+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:49.443 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:49.440+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:49.769 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:49.765+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:49.876 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:49.873+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:50.004 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:50.000+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:50.064 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:50.059+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:50.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:50.177+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:50.485 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:50.481+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:50.735 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:50.731+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:50.859 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:50.855+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:51.018 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:51.016+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:51.092 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:51.088+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:51.193 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:51.190+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:51.482 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:51.479+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:51.737 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:51.733+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:51.844 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:51.841+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:52.073 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:52.069+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:52.147 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:52.143+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:52.168 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:52.166+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:52.465 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:52.462+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:52.726 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:52.723+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:52.820 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:52.795+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:53.063 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:53.060+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:53.134 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:53.131+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:53.171 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:53.166+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:53.483 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:53.480+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:53.696 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:53.693+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:53.783 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:53.780+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:54.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:54.106+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:54.149 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:54.141+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:54.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:54.148+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:54.480 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:54.478+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:54.696 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:54.693+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:54.781 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:54.779+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:55.153 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:55.150+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:55.156 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:55.152+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:55.170 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:55.167+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:55.452 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:55.448+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:55.741 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:55.737+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:55.828 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:55.820+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:56.145 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:56.131+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:56.156 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:56.153+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:56.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:56.182+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:56.414 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:56.410+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:56.738 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:56.733+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:56.797 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:56.792+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:57.113 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:57.110+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:57.151 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:57.148+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:57.153 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:57.148+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:57.374 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:57.370+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:57.704 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:57.700+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:57.761 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:57.758+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:58.075 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:58.072+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:58.138 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:58.134+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:58.202 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:58.197+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:58.381 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:58.377+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:58.700 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:58.696+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:58.789 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:58.785+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:52:59.115 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:52:59.111+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:52:59.123 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:52:59.120+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:52:59.186 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:52:59.182+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:52:59.408 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:52:59.404+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:52:59.713 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:52:59.708+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:52:59.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:52:59.796+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:00.128 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:00.124+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:00.132 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:00.129+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:00.206 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:00.202+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:00.383 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:00.380+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:00.696 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:00.692+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:00.812 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:53:00.806+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:01.106 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:01.103+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:01.153 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:01.149+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:01.177 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:01.173+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:01.351 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:01.347+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:01.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:01.650+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:01.801 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:53:01.797+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:02.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:02.071+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:02.141 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:02.138+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:02.217 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:02.214+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:02.362 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:02.357+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:02.615 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:02.611+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:02.774 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:53:02.771+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:03.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:03.069+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:03.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:03.112+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:03.186 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:03.184+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:03.399 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:03.395+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:03.573 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:03.570+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:03.788 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:53:03.786+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:04.084 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:04.080+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:04.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:04.115+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:04.156 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:04.153+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:04.381 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:04.377+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:04.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:04.572+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:04.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:53:04.794+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:05.076 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:05.073+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:05.093 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:05.089+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:05.151 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:05.148+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:05.427 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:05.423+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:05.611 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:05.607+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:05.844 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:53:05.842+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:06.036 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:06.033+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:06.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:06.072+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:06.176 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:06.172+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:06.458 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:06.455+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:06.610 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:06.606+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:06.873 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:53:06.870+0000 1694f700 -1 osd.2 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.378695+0000 front 2019-12-11T23:51:51.381154+0000 (oldest deadline 2019-12-11T23:52:35.197284+0000) 2019-12-11T23:53:07.039 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:53:07.035+0000 1694f700 -1 osd.7 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.859153+0000 front 2019-12-11T23:51:51.173600+0000 (oldest deadline 2019-12-11T23:52:36.152329+0000) 2019-12-11T23:53:07.080 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:07.076+0000 1694f700 -1 osd.6 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.934691+0000 front 2019-12-11T23:51:50.172170+0000 (oldest deadline 2019-12-11T23:52:33.383058+0000) 2019-12-11T23:53:07.136 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:53:07.132+0000 1694f700 -1 osd.1 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.924932+0000 front 2019-12-11T23:51:49.813875+0000 (oldest deadline 2019-12-11T23:52:33.265786+0000) 2019-12-11T23:53:07.424 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:53:07.420+0000 1694f700 -1 osd.4 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:49.308879+0000 front 2019-12-11T23:51:48.784776+0000 (oldest deadline 2019-12-11T23:52:34.056737+0000) 2019-12-11T23:53:07.590 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:07.587+0000 1694f700 -1 osd.3 603 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:51:51.385093+0000 front 2019-12-11T23:51:51.253354+0000 (oldest deadline 2019-12-11T23:52:37.045552+0000) 2019-12-11T23:53:07.610 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:53:07.605+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 45sec e603 v603) 2019-12-11T23:53:41.836 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:41.816+0000 36790700 -1 osd.6 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:41.977 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:41.846+0000 30f85700 -1 osd.6 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.101 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:41.931+0000 31786700 -1 osd.6 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.144 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:41.932+0000 31786700 -1 osd.6 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.163 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:41.967+0000 34f8d700 -1 osd.6 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.284 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:42.239+0000 36790700 -1 osd.3 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.433 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:42.358+0000 32f89700 -1 osd.3 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.466 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:42.405+0000 31f87700 -1 osd.3 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.926 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:42.587+0000 32788700 -1 osd.3 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:42.986 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:53:42.785+0000 37f93700 -1 osd.3 607 build_incremental_map_msg missing incremental map 607 2019-12-11T23:53:49.744 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:53:49.739+0000 3478c700 -1 osd.6 609 build_incremental_map_msg missing incremental map 607 2019-12-11T23:54:02.744 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:02.743+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:03.781 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:03.778+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:04.786 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:04.784+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:05.752 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:05.751+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:06.656 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:06.652+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:06.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:06.795+0000 1694f700 -1 osd.2 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.839627+0000 front 2019-12-11T23:53:23.824755+0000 (oldest deadline 2019-12-11T23:54:06.094991+0000) 2019-12-11T23:54:06.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:06.797+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:07.004 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:07.001+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:07.658 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:07.654+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:07.792 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:07.789+0000 1694f700 -1 osd.2 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.839627+0000 front 2019-12-11T23:53:23.824755+0000 (oldest deadline 2019-12-11T23:54:06.094991+0000) 2019-12-11T23:54:07.793 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:07.790+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:07.881 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:07.879+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:07.986 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:07.983+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:08.652 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:08.642+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:08.784 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:08.780+0000 1694f700 -1 osd.2 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.839627+0000 front 2019-12-11T23:53:23.824755+0000 (oldest deadline 2019-12-11T23:54:06.094991+0000) 2019-12-11T23:54:08.784 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:08.782+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:08.853 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:08.851+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:08.978 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:08.976+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:09.080 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:09.077+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:09.665 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:09.658+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:09.837 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:09.831+0000 1694f700 -1 osd.2 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.839627+0000 front 2019-12-11T23:53:23.824755+0000 (oldest deadline 2019-12-11T23:54:06.094991+0000) 2019-12-11T23:54:09.848 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:09.834+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:09.866 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:09.864+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:10.013 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:10.011+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:10.052 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:10.048+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:10.676 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:10.672+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:10.813 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:10.810+0000 1694f700 -1 osd.2 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.839627+0000 front 2019-12-11T23:53:23.824755+0000 (oldest deadline 2019-12-11T23:54:06.094991+0000) 2019-12-11T23:54:10.814 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:10.811+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:10.857 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:10.853+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:11.012 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:11.009+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:11.014 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:11.011+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:11.675 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:11.672+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:11.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:11.795+0000 1694f700 -1 osd.2 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.839627+0000 front 2019-12-11T23:53:23.824755+0000 (oldest deadline 2019-12-11T23:54:06.094991+0000) 2019-12-11T23:54:11.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:11.796+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:11.818 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:11.815+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:12.026 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:12.022+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:12.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:12.046+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:12.706 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:12.704+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:12.798 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:12.796+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:12.837 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:12.833+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:13.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:12.998+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:13.040 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:13.037+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:13.694 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:13.691+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:13.839 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:13.837+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:13.849 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:13.846+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:14.027 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:14.023+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:14.065 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:14.062+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:14.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:14.740+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:14.793 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:14.792+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:14.873 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:14.870+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:14.964 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:14.962+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:15.018 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:15.015+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:15.076 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:15.069+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:15.699 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:15.697+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:15.778 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:15.777+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:15.872 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:15.869+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:15.986 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:15.984+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:16.032 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:16.029+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:16.077 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:16.074+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:16.665 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:16.662+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:16.791 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:16.789+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:16.872 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:16.869+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:17.035 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:17.032+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:17.077 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:17.074+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:17.130 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:17.127+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:17.645 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:17.642+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:17.790 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:17.788+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:17.906 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:17.902+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:18.053 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:18.050+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:18.078 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:18.071+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:18.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:18.129+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:18.630 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:18.627+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:18.822 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:18.820+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:18.902 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:18.899+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:19.045 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:19.043+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:19.047 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:19.046+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:19.055 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:19.052+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:19.126 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:19.123+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:19.585 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:19.582+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:19.815 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:19.813+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:19.934 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:19.930+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:20.037 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:20.034+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:20.096 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:20.094+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:20.099 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:20.097+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:20.149 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:20.142+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:20.580 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:20.577+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:20.862 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:20.860+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:20.903 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:20.899+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:21.025 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:21.021+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:21.131 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:21.128+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:21.131 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:21.129+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:21.189 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:21.187+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:21.621 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:21.617+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:21.888 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:21.886+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:21.889 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:21.886+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:22.014 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:22.009+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:22.148 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:22.144+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:22.150 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:22.146+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:22.153 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:22.150+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:22.575 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:22.571+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:22.940 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:22.933+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:22.941 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:22.938+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:23.063 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:23.058+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:23.128 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:23.126+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:23.143 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:23.138+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:23.144 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:23.141+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:23.553 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:23.550+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:23.918 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:23.914+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:23.923 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:23.919+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:24.038 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:24.033+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:24.092 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:24.089+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:24.179 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:24.175+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:24.179 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:24.176+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:24.532 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:24.531+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:24.890 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:24.888+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:24.910 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:24.909+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:25.047 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:25.044+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:25.060 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:25.057+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:25.162 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:25.159+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:25.165 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:25.162+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:25.546 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:25.544+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:25.882 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:25.881+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:25.941 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:25.938+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:26.033 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:26.030+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:26.047 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:26.045+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:26.161 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:26.157+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:26.162 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:26.159+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:26.554 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:26.552+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:26.881 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:26.879+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:26.906 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:26.903+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:27.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:27.044+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:27.095 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:27.091+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:27.123 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:27.120+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:27.124 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:27.122+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:27.600 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:27.598+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:27.891 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:27.886+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:27.909 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:27.907+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:28.090 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:28.088+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:28.100 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:28.097+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:28.143 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:28.139+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:28.146 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:28.143+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:28.565 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:28.563+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:28.942 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:28.940+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:28.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:28.952+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:29.107 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:29.106+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:29.120 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:29.117+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:29.145 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:29.141+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:29.145 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:29.142+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:29.538 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:29.536+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:29.945 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:29.940+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:29.956 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:29.953+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:30.084 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:30.081+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:30.155 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:30.153+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:30.181 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:30.178+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:30.182 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:30.179+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:30.589 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:30.587+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:30.969 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:30.967+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:30.984 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:30.981+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:31.110 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:31.108+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:31.115 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:31.114+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:31.199 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:31.195+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:31.199 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:31.197+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:31.639 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:31.637+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:32.000 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:31.996+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:32.007 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:32.001+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:32.118 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:32.113+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:32.135 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:32.132+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:32.183 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:32.175+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:32.183 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:32.177+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:32.634 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:32.632+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:32.962 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:32.960+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:32.964 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:32.962+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:33.108 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:33.105+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:33.156 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:33.154+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:33.194 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:33.192+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:33.195 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:33.193+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:33.620 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:33.617+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:33.961 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:33.959+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:34.008 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:34.003+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:34.081 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:34.078+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:34.172 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:34.169+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:34.200 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:34.198+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:34.202 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:34.199+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:34.627 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:34.624+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:34.981 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:34.978+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:35.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:35.035+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:35.082 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:35.079+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:35.194 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:35.191+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:35.196 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:35.192+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:35.196 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:35.194+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:35.662 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:35.659+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:35.980 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:35.979+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:36.010 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:36.007+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:36.042 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:36.039+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:36.198 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:36.194+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:36.199 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:36.196+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:36.220 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:36.217+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:36.674 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:36.672+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:37.012 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:37.009+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:37.032 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:37.029+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:37.050 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:37.048+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:37.167 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:37.164+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:37.168 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:37.166+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:37.214 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:37.212+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:37.672 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:37.664+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:37.846 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:54:37.840+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e609 v609) 2019-12-11T23:54:38.014 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:38.013+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:38.036 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:38.034+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:38.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:38.035+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:38.151 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:38.149+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:38.151 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:38.149+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:38.251 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:38.243+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:38.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:38.633+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:38.995 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:38.992+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:39.017 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:39.014+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:39.060 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:39.058+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:39.201 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:39.194+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:39.201 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:39.196+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:39.253 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:39.251+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:39.597 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:39.594+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:40.003 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:40.000+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:40.024 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:40.021+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:40.081 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:40.079+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:40.198 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:40.194+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:40.199 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:40.196+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:40.296 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:40.293+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:40.642 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:40.639+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:40.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:40.988+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:41.001 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:40.998+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:41.079 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:41.076+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:41.243 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:41.240+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:41.244 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:41.242+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:41.299 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:41.297+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:41.668 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:41.666+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:41.960 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:41.958+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:42.020 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:42.017+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:42.068 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:42.066+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:42.271 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:42.269+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:42.288 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:42.283+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:42.289 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:42.284+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:42.690 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:42.687+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:42.858 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:54:42.854+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e609 v609) 2019-12-11T23:54:42.957 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:42.953+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:43.009 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:43.001+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:43.117 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:43.115+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:43.271 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:43.269+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:43.316 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:43.314+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:43.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:43.315+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:43.718 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:43.715+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:43.930 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:43.927+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:44.053 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:44.050+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:44.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:44.110+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:44.322 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:44.320+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:44.355 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:44.352+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:44.355 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:44.353+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:44.682 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:44.679+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:44.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:44.931+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:45.068 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:45.065+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:45.130 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:45.129+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:45.342 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:45.340+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:45.345 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:45.344+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:45.349 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:45.346+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:45.687 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:45.684+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:45.891 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:45.888+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:46.107 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:46.101+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:46.162 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:46.160+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:46.323 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:46.321+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:46.323 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:46.321+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:46.341 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:46.331+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:46.695 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:46.692+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:46.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:46.934+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:47.145 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:47.143+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:47.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:47.150+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:47.314 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:47.305+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:47.322 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:47.319+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:47.323 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:47.321+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:47.729 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:47.725+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:47.874 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:54:47.872+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e609 v609) 2019-12-11T23:54:47.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:47.934+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:48.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:48.171+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:48.186 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:48.185+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:48.282 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:48.277+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:48.283 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:48.278+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:48.299 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:48.297+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:48.692 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:48.690+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:48.954 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:48.951+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:49.219 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:49.217+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:49.229 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:49.227+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:49.246 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:49.245+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:49.247 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:49.246+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:49.330 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:49.328+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:49.691 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:49.687+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:49.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:49.934+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:50.181 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:50.180+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:50.223 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:50.220+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:50.223 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:50.221+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:50.236 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:50.233+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:50.328 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:50.326+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:50.686 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:50.682+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:50.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:50.928+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:51.222 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:51.220+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:51.233 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:51.231+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:51.273 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:51.271+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:51.274 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:51.273+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:51.287 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:51.285+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:51.735 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:51.731+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:51.943 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:51.940+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:52.232 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:52.230+0000 1694f700 -1 osd.2 609 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:52.252 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:54:52.249+0000 1694f700 -1 osd.4 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:23.044974+0000 front 2019-12-11T23:53:23.221511+0000 (oldest deadline 2019-12-11T23:54:07.048623+0000) 2019-12-11T23:54:52.271 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:52.266+0000 1694f700 -1 osd.0 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:53:38.862393+0000 (oldest deadline 2019-12-11T23:54:18.862393+0000) 2019-12-11T23:54:52.271 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:52.269+0000 1694f700 -1 osd.0 609 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:52.304 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:54:52.302+0000 1694f700 -1 osd.1 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.008433+0000 front 2019-12-11T23:53:23.923185+0000 (oldest deadline 2019-12-11T23:54:06.808777+0000) 2019-12-11T23:54:52.756 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:54:52.753+0000 1694f700 -1 osd.6 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:24.717672+0000 front 2019-12-11T23:53:24.931776+0000 (oldest deadline 2019-12-11T23:54:06.365203+0000) 2019-12-11T23:54:52.944 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:54:52.941+0000 1694f700 -1 osd.7 609 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:53:22.971499+0000 front 2019-12-11T23:53:23.113543+0000 (oldest deadline 2019-12-11T23:54:08.741262+0000) 2019-12-11T23:54:52.958 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:54:52.950+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e609 v609) 2019-12-11T23:54:53.364 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:53.292+0000 1694f700 -1 osd.2 610 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:54:53.596 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:54:53.387+0000 1694f700 -1 osd.0 610 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e607) 2019-12-11T23:54:54.358 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:54:54.291+0000 1694f700 -1 osd.2 611 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120155 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e605) 2019-12-11T23:55:00.405 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:55:00.402+0000 1a89c700 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg recovery_unfound (PG_DAMAGED) 2019-12-11T23:55:23.496 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.454+0000 114ae700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:23.499 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.469+0000 35f8f700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:23.606 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.487+0000 38794700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:23.628 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.495+0000 35f8f700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:23.649 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.503+0000 3478c700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:23.697 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.503+0000 38794700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:23.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.539+0000 32788700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:23.797 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:23.634+0000 114ae700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:24.964 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:55:24.936+0000 31786700 -1 osd.3 615 build_incremental_map_msg missing incremental map 615 2019-12-11T23:55:39.259 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:23:26.192 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.263 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:23:26.197 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.277 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:23:26.210 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.288 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:23:26.221 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.290 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:23:26.195 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.299 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:23:26.205 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.561 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:23:26.494 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.580 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:23:26.513 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:55:39.635 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:23:26.541 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:17.070 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:17.068+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:18.087 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:18.085+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:19.117 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:19.116+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:20.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:20.141+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:21.181 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:21.180+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:22.228 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:22.227+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:23.201 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:23.200+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:24.201 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:24.200+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:24.227 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:24.221+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:25.193 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:25.192+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:25.206 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:25.198+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:26.229 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:26.225+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:26.231 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:26.228+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:26.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:26.245+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:27.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:27.160+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:27.266 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:27.261+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:27.267 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:27.264+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:27.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:27.300+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:28.203 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:28.200+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:28.261 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:28.258+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:28.263 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:28.260+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:28.334 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:28.330+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:28.769 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:28.765+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:29.039 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:29.036+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:29.085 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:29.078+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:29.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:29.229+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:29.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:29.233+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:29.252 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:29.249+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:29.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:29.383+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:29.741 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:29.738+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:30.086 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:30.083+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:30.092 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:30.089+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:30.233 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:30.229+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:30.237 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:30.231+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:30.239 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:30.235+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:30.384 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:30.382+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:30.765 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:30.761+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:31.072 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:31.068+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:31.146 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:31.142+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:31.233 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:31.230+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:31.236 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:31.234+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:31.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:31.273+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:31.413 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:31.410+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:31.776 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:31.774+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:32.075 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:32.071+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:32.199 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:32.195+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:32.287 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:32.284+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:32.289 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:32.286+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:32.290 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:32.289+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:32.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:32.431+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:32.821 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:32.818+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:33.057 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:33.053+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:33.195 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:33.192+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:33.310 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:33.306+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:33.311 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:33.309+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:33.311 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:33.310+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:33.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:33.435+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:33.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:33.780+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:34.046 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:34.042+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:34.195 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:34.191+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:34.267 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:34.261+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:34.267 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:34.263+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:34.311 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:34.308+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:34.412 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:34.409+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:34.758 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:34.752+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:35.069 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:35.064+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:35.232 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:35.229+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:35.233 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:35.230+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:35.235 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:35.232+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:35.360 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:35.356+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:35.425 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:35.422+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:35.731 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:35.728+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:36.088 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:36.084+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:36.239 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:36.234+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:36.244 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:36.241+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:36.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:36.243+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:36.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:36.393+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:36.474 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:36.471+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:36.770 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:36.766+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:37.083 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:37.078+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:37.247 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:37.244+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:37.289 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:37.285+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:37.289 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:37.288+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:37.402 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:37.398+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:37.440 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:37.437+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:37.817 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:37.813+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:38.047 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:38.044+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:38.249 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:38.246+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:38.336 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:38.332+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:38.336 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:38.335+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:38.423 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:38.419+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:38.490 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:38.487+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:38.864 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:38.860+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:39.075 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:39.072+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:39.243 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:39.239+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:39.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:39.377+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:39.383 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:39.380+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:39.434 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:39.430+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:39.535 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:39.532+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:39.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:39.898+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:40.061 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:40.057+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:40.204 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:40.200+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:40.365 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:40.362+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:40.366 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:40.364+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:40.413 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:40.410+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:40.587 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:40.583+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:40.904 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:40.899+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:41.063 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:41.058+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:41.194 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:41.190+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:41.366 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:41.363+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:41.368 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:41.366+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:41.409 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:41.406+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:41.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:41.549+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:41.871 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:41.868+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:42.048 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:42.046+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:42.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:42.209+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:42.420 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:42.409+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:42.422 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:42.420+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:42.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:42.460+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:42.592 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:42.588+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:42.859 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:42.856+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:43.069 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:43.065+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:43.227 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:43.224+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:43.428 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:43.424+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:43.428 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:43.427+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:43.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:43.476+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:43.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:43.598+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:43.861 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:43.860+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:44.081 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:44.078+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:44.218 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:44.215+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:44.446 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:44.443+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:44.446 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:44.444+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:44.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:44.475+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:44.637 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:44.628+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:44.906 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:44.896+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:45.072 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:45.070+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:45.265 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:45.262+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:45.430 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:45.427+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:45.434 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:45.432+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:45.481 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:45.475+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:45.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:45.664+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:45.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:45.900+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:46.050 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:46.047+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:46.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:46.316+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:46.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:46.455+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:46.463 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:46.460+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:46.466 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:46.464+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:46.715 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:46.712+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:46.894 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:46.891+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:47.102 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:47.100+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:47.329 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:47.323+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:47.432 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:47.428+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:47.432 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:47.430+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:47.484 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:47.480+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:47.683 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:47.679+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:47.858 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:47.856+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:48.122 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:48.120+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:48.371 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:48.368+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:48.412 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:48.409+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:48.415 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:48.414+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:48.498 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:48.494+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:48.688 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:48.685+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:48.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:48.845+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:49.127 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:49.125+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:49.377 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:49.374+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:49.468 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:49.464+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:49.468 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:49.467+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:49.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:49.476+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:49.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:49.648+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:49.829 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:49.825+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:50.174 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:50.170+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:50.399 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:50.395+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:50.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:50.423+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:50.430 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:50.428+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:50.525 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:50.522+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:50.642 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:50.639+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:50.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:50.783+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:51.150 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:51.147+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:51.432 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:51.429+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:51.446 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:51.443+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:51.447 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:51.445+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:51.514 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:51.511+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:51.629 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:51.626+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:51.771 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:51.768+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:52.180 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:52.177+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:52.424 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:52.421+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:52.433 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:52.430+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:52.436 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:52.434+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:52.532 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:52.530+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:52.654 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:52.650+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:52.733 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:52.731+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:53.178 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:53.174+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:53.416 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:53.412+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:53.417 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:53.415+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:53.468 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:53.465+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:53.496 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:24:40.430 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.548 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:24:40.482 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.549 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:53.546+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:53.552 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:24:40.485 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.577 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:24:40.482 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.583 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:24:40.516 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.589 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:24:40.494 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.630 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:53.628+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:53.646 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:24:40.579 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.655 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:24:40.588 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.680 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:24:40.585 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:56:53.697 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:53.695+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:54.168 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:54.165+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:54.398 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:54.394+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:54.399 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:54.396+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:54.485 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:54.483+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:54.579 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:54.576+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:54.656 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:54.652+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:54.661 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:54.658+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:55.141 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:55.138+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:55.412 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:55.407+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:55.414 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:55.410+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:55.473 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:55.471+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:55.564 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:55.561+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:55.672 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:55.669+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:55.683 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:55.681+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:56.190 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:56.186+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:56.421 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:56.419+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:56.424 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:56.423+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:56.448 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:56.445+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:56.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:56.600+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:56.636 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:56.634+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:56.674 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:56.671+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:57.206 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:57.204+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:57.426 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:57.423+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:57.467 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:57.462+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:57.468 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:57.465+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:57.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:57.575+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:57.687 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:57.683+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:57.688 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:57.685+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:58.170 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:58.166+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:58.459 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:58.455+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:58.477 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:56:58.471+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 47sec e615 v615) 2019-12-11T23:56:58.512 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:58.508+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:58.512 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:58.510+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:58.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:58.598+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:58.732 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:58.729+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:56:58.732 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:58.730+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:59.144 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:56:59.141+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:56:59.427 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:56:59.425+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:56:59.563 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:59.559+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:56:59.564 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:56:59.562+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:56:59.601 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:56:59.598+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:56:59.707 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:56:59.705+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:56:59.784 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:56:59.780+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:57:00.110 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:57:00.107+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:57:00.437 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:57:00.431+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:57:00.554 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:57:00.552+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:57:00.613 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:00.610+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:57:00.616 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:00.614+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:57:00.683 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:57:00.681+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:57:00.756 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:57:00.752+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:57:01.101 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:57:01.099+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:57:01.441 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:57:01.437+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:57:01.543 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:57:01.540+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:57:01.644 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:01.640+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:57:01.645 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:01.643+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:57:01.726 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:57:01.724+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:57:01.763 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:57:01.760+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:57:02.135 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:57:02.133+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:57:02.483 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:57:02.479+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:57:02.580 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:57:02.577+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:57:02.646 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:02.642+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:57:02.647 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:02.645+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:57:02.736 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:57:02.732+0000 1694f700 -1 osd.7 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:38.191008+0000 front 2019-12-11T23:55:38.285445+0000 (oldest deadline 2019-12-11T23:56:24.081656+0000) 2019-12-11T23:57:02.784 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:57:02.778+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:57:03.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:57:03.104+0000 1694f700 -1 osd.1 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:42.499537+0000 front 2019-12-11T23:55:42.505608+0000 (oldest deadline 2019-12-11T23:56:28.270839+0000) 2019-12-11T23:57:03.442 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:57:03.438+0000 1694f700 -1 osd.4 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.545242+0000 front 2019-12-11T23:55:43.529437+0000 (oldest deadline 2019-12-11T23:56:28.790313+0000) 2019-12-11T23:57:03.557 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:57:03.553+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 47sec e615 v615) 2019-12-11T23:57:03.571 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:57:03.569+0000 1694f700 -1 osd.2 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.993489+0000 front 2019-12-11T23:55:44.100424+0000 (oldest deadline 2019-12-11T23:56:26.871181+0000) 2019-12-11T23:57:03.648 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:03.636+0000 1694f700 -1 osd.6 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.066035+0000 front 2019-12-11T23:55:43.118211+0000 (oldest deadline 2019-12-11T23:56:25.894283+0000) 2019-12-11T23:57:03.651 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:03.646+0000 1694f700 -1 osd.6 615 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:57:03.789 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:57:03.787+0000 1694f700 -1 osd.3 615 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:55:43.991093+0000 front 2019-12-11T23:55:43.968042+0000 (oldest deadline 2019-12-11T23:56:28.269658+0000) 2019-12-11T23:57:04.645 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:04.644+0000 1694f700 -1 osd.6 616 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:57:05.728 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:57:05.713+0000 1694f700 -1 osd.6 617 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:120385 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs out=1131b,stat out=16b,read 0~21778] snapc 0=[] ondisk+read+known_if_redirected e615) 2019-12-11T23:57:26.387 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:57:26.332+0000 21452700 -1 reset not still connected to 0x10417580 2019-12-11T23:57:26.388 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:57:26.332+0000 21452700 -1 reset not still connected to 0x244be9e0 2019-12-11T23:57:33.865 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:57:33.214+0000 33f8b700 -1 osd.0 620 build_incremental_map_msg missing incremental map 620 2019-12-11T23:57:33.928 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:57:33.223+0000 33f8b700 -1 osd.0 620 build_incremental_map_msg missing incremental map 620 2019-12-11T23:57:34.425 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:57:34.415+0000 37f93700 -1 osd.0 621 build_incremental_map_msg missing incremental map 620 2019-12-11T23:58:01.799 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:01.797+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:02.230 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:02.226+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:02.530 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:02.526+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:02.637 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:02.635+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:02.795 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:02.792+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:02.838 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:02.834+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:02.961 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:02.958+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:03.213 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:03.211+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:03.486 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:03.482+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:03.670 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:03.668+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:03.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:03.775+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:03.881 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:03.878+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:03.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:03.970+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:04.199 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:04.191+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:04.461 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:04.457+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:04.649 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:04.647+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:04.765 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:04.761+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:04.878 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:04.874+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:04.940 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:04.939+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:05.186 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:05.184+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:05.479 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:05.475+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:05.680 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:05.676+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:05.783 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:05.781+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:05.919 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:05.916+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:05.991 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:05.988+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:06.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:06.163+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:06.491 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:06.488+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:06.647 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:06.639+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:06.773 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:06.770+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:06.911 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:06.907+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:06.996 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:06.994+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:07.162 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:07.159+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:07.469 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:07.465+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:07.681 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:07.678+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:07.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:07.790+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:07.948 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:07.944+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:08.048 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:08.046+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:08.129 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:08.126+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:08.489 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:08.486+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:08.733 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:08.731+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:08.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:08.759+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:08.779 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:08.761+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:08.970 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:08.967+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:09.023 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:09.020+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:09.101 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:09.098+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:09.507 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:09.505+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:09.712 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:09.709+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:09.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:09.731+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:09.736 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:09.734+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:10.020 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:10.017+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:10.067 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:10.063+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:10.102 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:10.099+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:10.530 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:10.528+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:10.705 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:10.702+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:10.706 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:10.704+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:10.725 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:10.723+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:11.016 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:11.013+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:11.079 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:11.076+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:11.099 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:11.097+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:11.514 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:11.512+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:11.678 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:11.675+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:11.679 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:11.677+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:11.711 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:11.708+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:12.060 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:12.057+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:12.090 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:12.088+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:12.094 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:12.091+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:12.488 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:12.486+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:12.634 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:12.632+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:12.635 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:12.634+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:12.720 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:12.717+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:13.057 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:13.055+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:13.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:13.062+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:13.074 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:13.071+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:13.522 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:13.519+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:13.641 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:13.639+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:13.642 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:13.641+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:13.732 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:13.730+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:14.062 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:14.060+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:14.096 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:14.093+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:14.113 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:14.112+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:14.504 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:14.500+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:14.642 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:14.639+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:14.643 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:14.640+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:14.787 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:14.785+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:15.053 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:15.051+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:15.069 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:15.067+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:15.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:15.148+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:15.470 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:15.468+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:15.681 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:15.674+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:15.681 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:15.677+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:15.831 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:15.829+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:16.034 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:16.032+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:16.068 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:16.066+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:16.132 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:16.130+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:16.524 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:16.521+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:16.692 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:16.689+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:16.694 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:16.693+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:16.793 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:16.788+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:17.006 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:17.003+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:17.028 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:17.026+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:17.117 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:17.115+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:17.494 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:17.491+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:17.718 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:17.715+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:17.718 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:17.716+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:17.786 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:17.784+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:17.975 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:17.972+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:18.066 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:18.064+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:18.116 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:18.115+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:18.525 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:18.522+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:18.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:18.756+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:18.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:18.757+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:18.761 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:18.760+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:18.979 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:18.977+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:19.112 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:19.110+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:19.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:19.171+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:19.550 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:19.547+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:19.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:19.719+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:19.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:19.732+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:19.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:19.733+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:20.013 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:20.009+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:20.160 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:20.155+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:20.161 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:20.160+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:20.519 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:20.516+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:20.715 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:20.713+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:20.753 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:20.752+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:20.754 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:20.753+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:21.003 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:21.001+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:21.136 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:21.133+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:21.187 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:21.184+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:21.558 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:21.554+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:21.704 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:21.701+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:21.752 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:21.745+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:21.753 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:21.749+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:22.046 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:22.043+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:22.170 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:22.168+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:22.213 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:22.210+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:22.570 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:22.568+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:22.731 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:22.729+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:22.769 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:22.766+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:22.770 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:22.768+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:23.085 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:23.079+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:23.172 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:23.169+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:23.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:23.221+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:23.538 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:23.535+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:23.744 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:23.741+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:23.821 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:23.806+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:23.821 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:23.808+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:24.124 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:24.120+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:24.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:24.152+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:24.232 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:24.230+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:24.570 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:24.568+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:24.709 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:24.703+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:24.836 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:24.834+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:24.841 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:24.837+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:25.128 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:25.124+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:25.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:25.126+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:25.177 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:25.175+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:25.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:25.207+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:25.593 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:25.591+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:25.681 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:25.678+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:25.816 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:25.812+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:25.816 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:25.814+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:26.145 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:26.142+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:26.147 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:26.143+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:26.147 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:26.145+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:26.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:26.195+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:26.585 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:26.583+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:26.706 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:26.703+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:26.854 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:26.852+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:26.855 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:26.853+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:27.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:27.122+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:27.160 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:27.156+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:27.161 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:27.158+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:27.232 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:27.229+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:27.581 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:27.579+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:27.737 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:27.734+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:27.862 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:27.858+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:27.862 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:27.860+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:28.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:28.107+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:28.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:28.110+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:28.113 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:28.111+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:28.220 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:28.217+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:28.577 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:28.574+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:28.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:28.735+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:28.850 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:28.848+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:28.851 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:28.849+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:29.155 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:29.153+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:29.165 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:29.154+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:29.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:29.155+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:29.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:29.180+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:29.588 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:29.586+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:29.716 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:29.713+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:29.845 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:29.843+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:29.847 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:29.845+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:30.157 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:30.154+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:30.176 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:30.173+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:30.176 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:30.174+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:30.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:30.207+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:30.578 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:30.570+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:30.721 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:30.719+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:30.883 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:30.880+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:30.883 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:30.882+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:31.186 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:31.182+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:31.186 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:31.184+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:31.202 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:31.200+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:31.210 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:31.207+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:31.603 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:31.600+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:31.705 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:31.702+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:31.845 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:31.842+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:31.846 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:31.844+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:32.153 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:32.150+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:32.153 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:32.151+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:32.171 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:32.168+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:32.254 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:32.251+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:32.621 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:32.618+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:32.710 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:32.707+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:32.858 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:32.855+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:32.859 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:32.857+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:33.141 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:33.138+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:33.142 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:33.140+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:33.225 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:33.222+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:33.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:33.266+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:33.614 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:33.610+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:33.665 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:33.662+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:33.875 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:33.872+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:33.875 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:33.873+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:34.100 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:34.097+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:34.101 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:34.098+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:34.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:34.209+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:34.261 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:34.258+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:34.656 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:34.653+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:34.703 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:34.701+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:34.860 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:34.857+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:34.860 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:34.859+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:35.145 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:35.143+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:35.145 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:35.144+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:35.204 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:35.202+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:35.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:35.220+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:35.711 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:35.704+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:35.724 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:35.714+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:35.815 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:35.813+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:35.816 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:35.815+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:36.143 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:36.141+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:36.146 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:36.144+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:36.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:36.193+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:36.209 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:36.207+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:36.692 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:36.690+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:36.756 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:36.751+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:36.804 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:36.802+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:36.805 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:36.803+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:37.147 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:37.144+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:37.147 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:37.145+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:37.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:37.198+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:37.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:37.220+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:37.738 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:37.735+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:37.762 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:37.759+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:37.764 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:37.762+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:37.766 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:37.764+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:38.148 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:38.145+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:38.148 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:38.146+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:38.181 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:38.173+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:38.195 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:38.192+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:38.733 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:38.731+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:38.777 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:38.774+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:38.777 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:38.775+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:38.789 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:58:38.784+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e621 v621) 2019-12-11T23:58:38.805 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:38.803+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:39.146 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:39.141+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:39.146 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:39.143+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:39.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:39.147+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:39.189 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:39.185+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:39.728 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:39.724+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:39.790 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:39.787+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:39.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:39.790+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:39.806 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:39.804+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:40.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:40.127+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:40.130 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:40.128+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:40.141 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:40.138+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:40.172 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:40.169+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:40.719 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:40.715+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:40.786 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:40.784+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:40.787 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:40.785+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:40.799 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:40.789+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:41.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:41.148+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:41.165 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:41.163+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:41.166 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:41.163+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:41.167 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:41.166+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:41.762 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:41.759+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:41.793 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:41.790+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:41.837 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:41.833+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:41.837 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:41.835+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:42.113 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:42.111+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:42.123 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:42.120+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:42.133 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:42.131+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:42.133 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:42.132+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:42.781 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:42.778+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:42.792 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:42.790+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:42.792 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:42.791+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:42.833 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:42.829+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:43.107 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:43.104+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:43.113 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-11T23:58:43.110+0000 1694f700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.759343+0000 front 2019-12-11T23:57:18.349433+0000 (oldest deadline 2019-12-11T23:58:02.427520+0000) 2019-12-11T23:58:43.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:43.141+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:43.146 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:43.144+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:43.752 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:43.749+0000 1694f700 -1 osd.0 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-11T23:57:28.298572+0000 (oldest deadline 2019-12-11T23:58:08.298572+0000) 2019-12-11T23:58:43.753 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-11T23:58:43.750+0000 1694f700 -1 osd.0 621 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e619) 2019-12-11T23:58:43.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-11T23:58:43.776+0000 1694f700 -1 osd.1 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.552533+0000 front 2019-12-11T23:57:20.513401+0000 (oldest deadline 2019-12-11T23:58:02.189667+0000) 2019-12-11T23:58:43.803 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-11T23:58:43.801+0000 1694f700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:17.761643+0000 front 2019-12-11T23:57:17.764861+0000 (oldest deadline 2019-12-11T23:58:01.848676+0000) 2019-12-11T23:58:43.854 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-11T23:58:43.851+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e621 v621) 2019-12-11T23:58:44.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:44.128+0000 1694f700 -1 osd.2 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:20.740406+0000 front 2019-12-11T23:57:20.713070+0000 (oldest deadline 2019-12-11T23:58:01.783893+0000) 2019-12-11T23:58:44.131 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:44.130+0000 1694f700 -1 osd.2 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:58:44.159 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-11T23:58:44.157+0000 1694f700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-11T23:57:16.485888+0000 front 2019-12-11T23:57:16.027710+0000 (oldest deadline 2019-12-11T23:58:01.850089+0000) 2019-12-11T23:58:45.225 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-11T23:58:45.196+0000 1694f700 -1 osd.2 622 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120600 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e621) 2019-12-11T23:59:15.442 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyFromHeader (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:59:19.659 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyFromHeader404s (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:59:29.185 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:27:16.118 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.197 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:27:16.131 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.203 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:27:16.135 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.207 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:27:16.140 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.222 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:27:16.127 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.245 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:27:16.151 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.421 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:27:16.355 9281== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.438 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:27:16.370 9282== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:29.549 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:27:16.454 8846== Warning: unimplemented fcntl command: 1036 2019-12-11T23:59:49.325 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:59:49.324+0000 1694f700 -1 osd.3 626 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:120784 7.3 7:ca9a12ea:::2019-12-11-23-14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673-%e3%9c%84%e5%98%88%e3%9c%86%eb%8c%8d%e0%a4%80%e5%98%86%e1%a0%84%ed%88%8d%e1%a0%84%ed%88%90%e0%a4%81%e1%a0%81%e0%a4%83%e5%98%88%e9%90%8a:head [append 0~736 in=736b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e625) 2019-12-11T23:59:50.294 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:59:50.293+0000 1694f700 -1 osd.3 626 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120784 7.3 7:ca9a12ea:::2019-12-11-23-14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673-%e3%9c%84%e5%98%88%e3%9c%86%eb%8c%8d%e0%a4%80%e5%98%86%e1%a0%84%ed%88%8d%e1%a0%84%ed%88%90%e0%a4%81%e1%a0%81%e0%a4%83%e5%98%88%e9%90%8a:head [append 0~736 in=736b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e625) 2019-12-11T23:59:51.273 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-11T23:59:51.271+0000 1694f700 -1 osd.3 626 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:120787 3.22 3:46e2c230:::.dir.14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673:head [call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=187b] snapc 0=[] ondisk+write+known_if_redirected e625) 2019-12-11T23:59:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:testCopyNoDestinationHeader (test.functional.tests.TestFileUTF8) ... ok 2019-12-11T23:59:59.808 INFO:teuthology.orchestra.run.smithi079.stderr:testDelete (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:00:00.833 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteOfFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:00:06.643 INFO:teuthology.orchestra.run.smithi079.stderr:testDeleteThen404s (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:00:08.253 INFO:teuthology.orchestra.run.smithi079.stderr:testEtagResponse (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:00:08.652 INFO:teuthology.orchestra.run.smithi079.stderr:testEtagWayoff (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:00:12.938 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:00:12.917+0000 31f87700 -1 osd.0 626 build_incremental_map_msg missing incremental map 620 2019-12-12T00:00:21.314 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:21.311+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:21.344 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:21.340+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:21.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:21.399+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:22.354 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:22.352+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:22.356 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:22.354+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:22.405 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:22.401+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:23.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:23.277+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:23.334 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:23.331+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:23.391 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:23.389+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:23.446 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:23.442+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:24.070 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:24.064+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:24.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:24.265+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:24.353 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:24.351+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:24.388 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:24.385+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:24.486 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:24.482+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:24.955 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:24.952+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:25.045 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:25.043+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:25.248 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:25.222+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:25.355 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:25.353+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:25.389 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:25.386+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:25.453 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:25.451+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:25.966 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:25.963+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:26.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:26.032+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:26.246 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:26.242+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:26.329 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:26.326+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:26.394 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:26.391+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:26.427 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:26.422+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:26.975 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:26.972+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:27.011 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:27.007+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:27.230 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:27.227+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:27.316 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:27.314+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:27.351 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:27.348+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:27.461 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:27.458+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:27.941 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:27.939+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:27.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:27.971+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:28.284 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:28.280+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:28.287 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:28.284+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:28.387 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:28.385+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:28.475 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:28.472+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:28.979 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:28.976+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:29.021 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:29.017+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:29.275 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:29.272+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:29.301 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:29.298+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:29.439 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:29.436+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:29.501 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:29.498+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:30.004 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:30.002+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:30.020 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:30.017+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:30.248 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:30.246+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:30.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:30.331+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:30.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:30.476+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:30.530 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:30.526+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:30.995 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:30.992+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:31.038 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:31.035+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:31.228 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:31.225+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:31.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:31.331+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:31.474 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:31.470+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:31.539 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:31.536+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:32.024 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:32.020+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:32.027 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:32.024+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:32.277 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:32.275+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:32.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:32.317+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:32.453 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:32.451+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:32.504 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:32.500+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:33.024 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:33.021+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:33.051 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:33.048+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:33.282 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:33.279+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:33.283 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:33.279+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:33.438 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:33.435+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:33.472 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:33.468+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:34.054 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:34.051+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:34.063 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:34.060+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:34.311 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:34.309+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:34.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:34.332+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:34.431 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:34.427+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:34.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:34.453+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:35.027 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:35.024+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:35.085 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:35.082+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:35.274 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:35.271+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:35.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:35.354+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:35.423 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:35.420+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:35.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:35.466+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:36.003 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:36.000+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:36.129 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:36.126+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:36.303 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:36.300+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:36.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:36.331+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:36.397 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:36.394+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:36.454 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:36.450+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:37.007 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:37.005+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:37.169 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:37.165+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:37.332 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:37.328+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:37.375 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:37.372+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:37.415 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:37.412+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:37.429 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:37.425+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:38.040 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:38.037+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:38.196 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:38.192+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:38.313 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:38.311+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:38.386 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:38.383+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:38.387 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:38.384+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:38.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:38.448+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:39.019 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:39.016+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:39.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:39.149+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:39.357 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:39.355+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:39.364 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:39.362+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:39.393 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:39.389+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:39.416 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:39.413+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:40.015 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:40.012+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:40.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:40.152+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:40.333 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:40.331+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:40.367 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:40.364+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:40.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:40.411+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:40.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:40.422+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:41.003 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:40.999+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:41.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:41.154+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:41.344 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:41.342+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:41.351 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:41.349+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:41.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:41.448+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:41.468 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:41.457+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:42.032 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:42.029+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:42.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:42.150+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:42.375 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:42.372+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:42.387 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:42.385+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:42.450 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:42.432+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:42.456 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:42.452+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:42.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:42.982+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:43.176 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:43.173+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:43.349 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:43.344+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:43.355 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:43.349+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:43.453 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:43.450+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:43.461 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:43.457+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:43.954 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:43.952+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:44.210 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:44.207+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:44.313 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:44.311+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:44.388 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:44.385+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:44.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:44.438+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:44.498 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:44.495+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:44.950 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:44.947+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:45.199 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:45.195+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:45.339 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:45.338+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:45.380 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:45.378+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:45.468 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:45.463+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:45.518 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:45.514+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:45.995 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:45.992+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:46.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:46.220+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:46.314 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:46.311+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:46.395 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:46.392+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:46.455 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:46.451+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:46.550 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:46.547+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:46.989 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:46.986+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:47.222 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:47.219+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:47.323 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:47.321+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:47.375 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:47.372+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:47.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:47.498+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:47.535 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:47.531+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:47.946 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:47.944+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:48.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:48.221+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:48.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:48.350+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:48.358 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:48.357+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:48.505 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:48.501+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:48.552 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:48.547+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:48.998 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:48.996+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:49.258 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:49.254+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:49.311 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:49.309+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:49.354 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:49.351+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:49.503 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:49.500+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:49.565 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:49.562+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:50.019 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:50.016+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:50.298 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:50.295+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:50.306 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:50.304+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:50.362 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:50.360+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:50.521 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:50.518+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:50.601 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:50.598+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:51.057 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:51.053+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:51.276 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:51.273+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:51.336 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:51.333+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:51.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:51.348+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:51.543 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:51.539+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:51.570 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:51.567+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:52.060 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:52.057+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:52.271 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:52.267+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:52.327 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:52.324+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:52.327 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:52.324+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:52.567 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:52.563+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:52.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:52.584+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:53.072 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:53.069+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:53.267 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:53.264+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:53.294 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:53.291+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:00:53.298 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:53.295+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:53.521 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:00:53.517+0000 1694f700 -1 osd.6 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:34.881614+0000 front 2019-12-11T23:59:40.917772+0000 (oldest deadline 2019-12-12T00:00:20.720399+0000) 2019-12-12T00:00:53.598 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:00:53.595+0000 1694f700 -1 osd.7 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.348361+0000 front 2019-12-11T23:59:38.360181+0000 (oldest deadline 2019-12-12T00:00:22.974953+0000) 2019-12-12T00:00:54.061 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:00:54.057+0000 1694f700 -1 osd.1 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.648743+0000 front 2019-12-11T23:59:38.837827+0000 (oldest deadline 2019-12-12T00:00:24.470149+0000) 2019-12-12T00:00:54.295 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:00:54.292+0000 1694f700 -1 osd.4 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:38.638124+0000 front 2019-12-11T23:59:38.540785+0000 (oldest deadline 2019-12-12T00:00:23.816684+0000) 2019-12-12T00:00:54.295 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:00:54.293+0000 1694f700 -1 osd.3 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:37.197751+0000 front 2019-12-11T23:59:37.332377+0000 (oldest deadline 2019-12-12T00:00:21.238370+0000) 2019-12-12T00:00:54.311 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:00:54.302+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 42sec e626 v626) 2019-12-12T00:00:54.320 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:00:54.315+0000 1694f700 -1 osd.2 626 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-11T23:59:40.901968+0000 front 2019-12-11T23:59:39.035258+0000 (oldest deadline 2019-12-12T00:00:20.675322+0000) 2019-12-12T00:01:14.715 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:29:01.648 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:14.747 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:29:01.680 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:14.773 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:29:01.706 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:14.779 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:29:01.712 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:14.824 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:29:01.730 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:14.828 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:29:01.734 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:14.858 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:29:01.792 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:14.937 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:29:01.870 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:15.023 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:29:01.926 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:01:28.875 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:01:28.684+0000 21452700 -1 reset not still connected to 0x1d63bc10 2019-12-12T00:01:30.590 INFO:teuthology.orchestra.run.smithi079.stderr:testFileCreate (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:01:34.664 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:01:34.654+0000 124b0700 -1 osd.0 632 build_incremental_map_msg missing incremental map 632 2019-12-12T00:01:34.772 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:01:34.718+0000 3578e700 -1 osd.0 632 build_incremental_map_msg missing incremental map 632 2019-12-12T00:01:35.069 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:01:34.828+0000 36f91700 -1 osd.0 632 build_incremental_map_msg missing incremental map 632 2019-12-12T00:01:36.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:01:36.713+0000 114ae700 -1 osd.0 633 build_incremental_map_msg missing incremental map 632 2019-12-12T00:01:37.722 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:01:37.664+0000 32f89700 -1 osd.0 633 build_incremental_map_msg missing incremental map 632 2019-12-12T00:01:40.854 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:01:40.853+0000 34f8d700 -1 osd.0 633 build_incremental_map_msg missing incremental map 632 2019-12-12T00:01:41.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:01:41.293+0000 31f87700 -1 osd.3 633 build_incremental_map_msg missing incremental map 607 2019-12-12T00:01:41.751 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:01:41.714+0000 31f87700 -1 osd.3 633 build_incremental_map_msg missing incremental map 615 2019-12-12T00:01:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:testGetContentType (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:01:53.204 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:01:53.201+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:01:53.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:01:53.274+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:01:54.161 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:01:54.159+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:01:54.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:01:54.255+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:01:55.024 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:01:55.021+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:01:55.131 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:01:55.128+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:01:55.139 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:01:55.136+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:01:55.254 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:01:55.250+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:01:55.269 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:01:55.265+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:01:56.061 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:01:56.057+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:01:56.134 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:01:56.132+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:01:56.180 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:01:56.177+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:01:56.223 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:01:56.220+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:01:56.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:01:56.277+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:01:57.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:01:57.107+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:01:57.110 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:01:57.107+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:01:57.163 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:01:57.162+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:01:57.219 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:01:57.217+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:01:57.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:01:57.259+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:01:58.074 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:01:58.071+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:01:58.173 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:01:58.156+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:01:58.173 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:01:58.156+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:01:58.178 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:01:58.176+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:01:58.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:01:58.306+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:01:59.094 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:01:59.091+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:01:59.160 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:01:59.157+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:01:59.176 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:01:59.174+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:01:59.206 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:01:59.204+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:01:59.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:01:59.280+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:00.058 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:00.055+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:00.163 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:00.161+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:00.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:00.167+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:00.189 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:00.187+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:00.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:00.249+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:01.081 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:01.079+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:01.135 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:01.132+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:01.196 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:01.194+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:01.197 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:01.193+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:01.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:01.286+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:02.084 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:02.078+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:02.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:02.130+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:02.181 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:02.180+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:02.236 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:02.234+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:02.275 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:02.273+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:03.086 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:03.084+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:03.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:03.123+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:03.225 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:03.223+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:03.282 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:03.280+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:03.286 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:03.284+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:04.116 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:04.114+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:04.161 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:04.160+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:04.197 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:04.195+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:04.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:04.258+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:04.296 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:04.293+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:04.658 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:04.656+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:05.096 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:05.094+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:05.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:05.153+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:05.237 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:05.235+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:05.279 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:05.277+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:05.287 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:05.285+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:05.617 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:05.615+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:06.094 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:06.092+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:06.152 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:06.150+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:06.248 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:06.246+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:06.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:06.281+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:06.304 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:06.302+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:06.599 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:06.597+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:07.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:07.108+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:07.112 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:07.110+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:07.259 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:07.257+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:07.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:07.258+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:07.284 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:07.282+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:07.567 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:07.566+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:08.080 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:08.077+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:08.142 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:08.141+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:08.260 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:08.259+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:08.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:08.298+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:08.325 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:08.324+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:08.595 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:08.593+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:09.085 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:09.083+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:09.191 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:09.189+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:09.231 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:09.229+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:09.303 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:09.299+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:09.354 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:09.349+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:09.607 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:09.606+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:10.092 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:10.090+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:10.172 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:10.169+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:10.233 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:10.229+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:10.336 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:10.334+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:10.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:10.390+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:10.630 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:10.628+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:10.632 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:10.630+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:11.130 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:11.128+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:11.190 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:11.187+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:11.217 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:11.214+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:11.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:11.309+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:11.390 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:11.388+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:11.588 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:11.584+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:11.588 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:11.586+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:12.094 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:12.091+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:12.199 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:12.195+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:12.199 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:12.196+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:12.338 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:12.335+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:12.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:12.391+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:12.555 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:12.551+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:12.559 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:12.553+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:13.125 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:13.122+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:13.177 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:13.174+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:13.204 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:13.199+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:13.324 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:13.319+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:13.324 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:13.322+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:13.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:13.405+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:13.518 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:13.515+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:13.529 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:13.518+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:14.142 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:14.140+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:14.146 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:14.143+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:14.231 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:14.229+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:14.299 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:14.296+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:14.301 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:14.300+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:14.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:14.369+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:14.483 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:14.480+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:14.484 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:14.481+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:15.123 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:15.121+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:15.158 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:15.154+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:15.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:15.185+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:15.261 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:15.257+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:15.262 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:15.259+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:15.367 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:15.364+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:15.499 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:15.496+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:15.510 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:15.501+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:16.130 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:16.128+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:16.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:16.140+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:16.168 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:16.165+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:16.272 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:16.270+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:16.273 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:16.271+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:16.322 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:16.320+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:16.521 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:16.519+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:16.523 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:16.521+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:17.125 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:17.122+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:17.145 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:17.143+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:17.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:17.194+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:17.237 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:17.234+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:17.237 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:17.236+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:17.349 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:17.347+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:17.552 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:17.548+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:17.553 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:17.551+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:18.090 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:18.087+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:18.160 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:18.158+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:18.214 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:18.211+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:18.281 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:18.278+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:18.281 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:18.279+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:18.307 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:18.304+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:18.542 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:18.540+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:18.543 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:18.542+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:19.126 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:19.124+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:19.187 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:19.185+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:19.190 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:19.188+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:19.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:19.277+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:19.288 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:19.286+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:19.289 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:19.287+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:19.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:19.506+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:19.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:19.508+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:20.088 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:20.086+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:20.171 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:20.164+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:20.172 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:20.164+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:20.284 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:20.281+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:20.310 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:20.304+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:20.311 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:20.305+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:20.464 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:20.462+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:20.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:20.463+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:21.092 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:21.089+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:21.151 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:21.148+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:21.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:21.209+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:21.293 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:21.291+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:21.317 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:21.315+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:21.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:21.316+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:21.454 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:21.452+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:21.454 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:21.453+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:22.071 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:22.067+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:22.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:22.138+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:22.257 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:22.254+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:22.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:22.342+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:22.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:22.350+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:22.354 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:22.352+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:22.431 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:22.428+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:22.432 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:22.430+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:23.073 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:23.071+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:23.123 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:23.119+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:23.252 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:23.245+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:23.352 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:23.348+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:23.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:23.350+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:23.372 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:23.370+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:23.473 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:23.471+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:23.476 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:23.475+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:24.087 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:24.085+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:24.119 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:24.116+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:24.225 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:24.222+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:24.355 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:24.353+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:24.355 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:24.353+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:24.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:24.389+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:24.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:24.501+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:24.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:24.502+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:24.806 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:24.803+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e633 v633) 2019-12-12T00:02:25.118 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:25.116+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:25.145 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:25.134+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:25.241 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:25.239+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:25.313 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:25.310+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:25.314 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:25.311+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:25.392 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:25.389+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:25.549 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:25.544+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:25.549 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:25.546+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:26.153 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:26.150+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:26.176 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:26.173+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:26.301 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:26.290+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:26.354 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:26.352+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:26.356 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:26.354+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:26.406 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:26.403+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:26.538 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:26.535+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:26.540 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:26.538+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:27.157 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:27.154+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:27.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:27.173+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:27.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:27.315+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:27.318 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:27.316+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:27.340 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:27.337+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:27.452 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:27.449+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:27.517 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:27.513+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:27.517 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:27.515+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:28.137 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:28.133+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:28.146 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:28.144+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:28.295 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:28.291+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:28.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:28.293+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:28.352 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:28.349+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:28.425 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:28.422+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:28.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:28.546+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:28.551 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:28.547+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:29.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:29.112+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:29.133 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:29.128+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:29.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:29.309+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:29.313 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:29.310+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:29.389 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:29.377+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:29.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:29.417+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:29.575 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:29.570+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:29.575 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:29.572+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:29.822 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:29.818+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e633 v633) 2019-12-12T00:02:30.133 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:30.130+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:30.149 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:30.145+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:30.306 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:30.303+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:30.309 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:30.306+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:30.344 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:30.341+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:30.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:30.391+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:30.587 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:30.584+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:30.589 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:30.587+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:31.128 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:31.124+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:31.131 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:31.128+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:31.343 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:31.341+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:31.359 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:31.356+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:31.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:31.359+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:31.367 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:31.365+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:31.563 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:31.561+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:31.565 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:31.563+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:32.113 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:32.111+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:32.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:32.180+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:32.362 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:32.360+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:32.365 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:32.363+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:32.367 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:32.366+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:32.400 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:32.397+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:32.615 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:32.612+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:32.616 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:32.614+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:33.070 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:33.067+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:33.196 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:33.193+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:33.351 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:33.348+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:33.358 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:33.356+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:33.419 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:33.416+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:33.419 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:33.418+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:33.636 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:33.633+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:33.637 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:33.636+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:34.117 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:34.115+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:34.180 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:34.178+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:34.365 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:34.361+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:34.402 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:34.397+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:34.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:34.464+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:34.481 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:34.468+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:34.668 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:34.662+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:34.668 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:34.664+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:34.835 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:34.829+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e633 v633) 2019-12-12T00:02:35.105 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:35.102+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:35.138 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:35.136+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:35.369 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:35.366+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:35.409 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:35.406+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:35.441 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:35.438+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:35.441 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:35.439+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:35.637 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:35.634+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:35.639 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:35.636+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:36.096 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:36.094+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:36.113 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:36.110+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:36.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:36.342+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:36.400 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:36.394+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:36.400 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:36.396+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:36.442 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:36.439+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:36.672 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:36.671+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:36.673 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:36.672+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:37.126 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:37.124+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:37.140 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:37.137+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:37.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:37.351+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:37.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:37.352+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:37.374 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:37.371+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:37.482 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:37.477+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:37.640 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:37.636+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:37.640 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:37.638+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:38.083 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:38.081+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:38.104 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:38.101+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:38.395 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:38.392+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:38.404 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:38.402+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:38.413 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:38.406+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:38.453 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:38.451+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:38.651 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:38.644+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:38.651 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:38.645+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:39.068 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:39.066+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:39.119 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:39.115+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:39.419 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:39.415+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:39.426 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:39.423+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:39.426 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:39.424+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:39.484 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:39.482+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:39.685 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:39.683+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:39.686 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:39.685+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:39.847 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:39.842+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e633 v633) 2019-12-12T00:02:40.073 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:40.071+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:40.155 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:40.153+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:40.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:40.439+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:40.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:40.454+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:40.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:40.455+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:40.519 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:40.517+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:40.706 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:40.704+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:40.707 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:40.705+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:41.105 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:41.104+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:41.167 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:41.163+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:41.401 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:41.398+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:41.479 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:41.476+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:41.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:41.489+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:41.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:41.490+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:41.719 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:41.717+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:41.721 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:41.720+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:42.082 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:42.080+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:42.178 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:42.176+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:42.442 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:42.440+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:42.493 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:42.487+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:42.496 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:42.494+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:42.496 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:42.495+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:42.759 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:42.753+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:42.759 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:42.754+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:43.107 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:43.105+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:43.142 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:43.138+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:43.402 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:43.400+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:43.475 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:43.472+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:43.537 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:43.534+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:43.538 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:43.536+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:43.761 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:43.759+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:43.764 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:43.763+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:44.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:44.134+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:44.192 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:44.190+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:44.414 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:44.411+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:44.537 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:44.529+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:44.537 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:44.526+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:44.537 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:44.527+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:44.777 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:44.774+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:44.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:44.777+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:44.859 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:44.856+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e633 v633) 2019-12-12T00:02:45.130 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:45.128+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:45.239 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:45.238+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:45.442 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:45.433+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:45.504 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:45.501+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:45.515 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:45.513+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:45.518 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:45.517+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:45.771 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:45.769+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:45.774 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:45.773+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:46.121 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:46.106+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:46.239 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:46.231+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:46.457 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:46.453+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:46.478 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:46.471+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:46.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:46.476+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:46.480 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:46.477+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:46.802 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:46.799+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:46.803 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:46.802+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:47.159 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:47.156+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:47.211 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:47.208+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:47.438 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:47.435+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:47.445 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:47.443+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:47.466 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:47.464+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:47.467 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:47.466+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:47.809 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:47.806+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:47.812 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:47.810+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:48.208 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:48.206+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:48.265 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:48.262+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:48.398 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:48.395+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:48.464 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:48.462+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:48.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:48.465+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:48.469 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:48.466+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:48.819 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:48.818+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:48.820 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:48.819+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:49.246 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:02:49.244+0000 1694f700 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:10.553751+0000 front 2019-12-12T00:01:10.332707+0000 (oldest deadline 2019-12-12T00:01:54.960789+0000) 2019-12-12T00:02:49.269 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:02:49.267+0000 1694f700 -1 osd.1 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.309395+0000 front 2019-12-12T00:01:11.472112+0000 (oldest deadline 2019-12-12T00:01:54.795556+0000) 2019-12-12T00:02:49.420 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:49.417+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:49.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:02:49.473+0000 1694f700 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:11.305201+0000 front 2019-12-12T00:01:11.194814+0000 (oldest deadline 2019-12-12T00:01:52.846023+0000) 2019-12-12T00:02:49.490 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:49.487+0000 1694f700 -1 osd.2 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:06.577416+0000 front 2019-12-12T00:01:06.576550+0000 (oldest deadline 2019-12-12T00:01:52.308318+0000) 2019-12-12T00:02:49.491 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:49.489+0000 1694f700 -1 osd.2 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:49.872 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:49.870+0000 1694f700 -1 osd.0 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:01:29.684539+0000 (oldest deadline 2019-12-12T00:02:09.684539+0000) 2019-12-12T00:02:49.874 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:49.871+0000 1694f700 -1 osd.0 633 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e630) 2019-12-12T00:02:49.911 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:49.908+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e633 v633) 2019-12-12T00:02:50.394 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:02:50.391+0000 1694f700 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:01:09.375840+0000 front 2019-12-12T00:01:09.346241+0000 (oldest deadline 2019-12-12T00:01:54.605247+0000) 2019-12-12T00:02:50.851 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:02:50.850+0000 34f8d700 -1 osd.0 634 build_incremental_map_msg missing incremental map 632 2019-12-12T00:02:51.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:51.096+0000 1694f700 -1 osd.2 634 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:52.181 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:02:52.178+0000 1694f700 -1 osd.2 635 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121159 6.f 6:fd8078d5:root::%e0%a4%81%e1%a0%83%e3%9c%83%e0%a4%83%eb%8c%8b%eb%8c%8c%eb%8c%8d%eb%8c%8c%e3%9c%86%e5%98%88%e2%98%83%e8%94%8b%e8%94%8b%e0%a4%80%eb%8c%8d:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e633) 2019-12-12T00:02:52.412 INFO:teuthology.orchestra.run.smithi079.stderr:testGetOnFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:02:54.754 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:54.749+0000 1a89c700 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg recovery_unfound (PG_DAMAGED) 2019-12-12T00:02:59.931 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:02:59.920+0000 1a89c700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 2 pgs recovery_unfound (PG_DAMAGED) 2019-12-12T00:03:15.239 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:31:02.164 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.244 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:31:02.178 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.261 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:31:02.194 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.265 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:31:02.198 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.282 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:31:02.188 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.296 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:31:02.199 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.504 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:31:02.431 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.507 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:31:02.412 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:15.529 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:31:02.463 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:03:41.746 INFO:teuthology.orchestra.run.smithi079.stderr:testHead (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:03:42.750 INFO:teuthology.orchestra.run.smithi079.stderr:testHeadOnFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:04:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:testMetadataOnPost (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:04:10.539 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:10.535+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:10.829 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:10.826+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:11.009 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:11.005+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:11.549 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:11.545+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:11.565 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:11.561+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:11.842 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:11.837+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:11.988 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:11.983+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:12.560 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:12.557+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:12.586 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:12.583+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:12.846 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:12.843+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:12.889 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:12.887+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:13.034 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:13.031+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:13.539 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:13.530+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:13.624 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:13.622+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:13.852 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:13.848+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:13.870 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:13.868+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:14.051 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:14.048+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:14.563 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:14.559+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:14.578 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:14.576+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:14.817 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:14.816+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:14.870 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:14.868+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:15.089 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:15.087+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:15.553 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:15.551+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:15.558 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:15.555+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:15.789 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:15.787+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:15.841 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:15.839+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:15.905 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:15.902+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:16.098 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:16.094+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:16.530 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:16.525+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:16.533 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:16.531+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:16.792 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:16.789+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:16.836 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:16.833+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:16.871 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:16.868+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:17.106 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:17.105+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:17.502 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:17.499+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:17.557 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:17.555+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:17.820 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:17.818+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:17.827 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:17.824+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:17.844 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:17.841+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:18.133 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:18.130+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:18.543 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:18.540+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:18.553 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:18.548+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:18.782 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:18.779+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:18.862 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:18.860+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:18.882 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:18.880+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:19.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:19.161+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:19.558 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:19.554+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:19.594 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:19.592+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:19.756 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:19.753+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:19.830 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:19.824+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:19.913 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:19.908+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:20.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:20.192+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:20.563 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:20.560+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:20.605 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:20.602+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:20.748 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:20.746+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:20.853 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:20.850+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:20.869 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:20.866+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:21.194 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:21.190+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:21.530 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:21.527+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:21.568 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:21.565+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:21.791 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:21.789+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:21.812 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:21.810+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:21.919 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:21.917+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:22.171 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:22.168+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:22.557 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:22.553+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:22.576 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:22.572+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:22.787 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:22.782+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:22.796 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:22.794+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:22.944 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:22.941+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:23.127 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:23.124+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:23.537 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:23.534+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:23.557 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:23.554+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:23.761 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:23.758+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:23.834 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:23.831+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:23.975 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:23.973+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:24.180 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:24.177+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:24.518 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:24.516+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:24.579 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:24.577+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:24.797 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:24.795+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:24.853 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:24.849+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:24.937 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:24.934+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:25.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:25.217+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:25.540 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:25.538+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:25.559 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:25.555+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:25.789 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:25.786+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:25.876 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:25.870+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:25.917 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:25.914+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:26.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:26.259+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:26.540 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:26.536+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:26.570 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:26.567+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:26.820 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:26.819+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:26.897 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:26.895+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:26.914 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:26.913+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:27.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:27.283+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:27.498 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:27.495+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:27.589 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:27.586+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:27.868 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:27.866+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:27.939 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:27.936+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:27.954 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:27.950+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:28.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:28.265+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:28.504 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:28.501+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:28.633 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:28.630+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:28.827 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:28.825+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:28.920 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:28.915+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:28.991 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:28.988+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:29.282 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:29.279+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:29.529 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:29.526+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:29.614 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:29.611+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:29.858 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:29.855+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:29.887 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:29.884+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:30.003 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:30.000+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:30.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:30.308+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:30.565 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:30.562+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:30.621 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:30.618+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:30.851 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:30.848+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:30.854 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:30.850+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:31.000 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:30.995+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:31.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:31.274+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:31.550 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:31.547+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:31.631 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:31.629+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:31.848 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:31.846+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:31.858 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:31.856+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:32.014 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:32.011+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:32.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:32.280+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:32.510 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:32.502+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:32.583 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:32.581+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:32.829 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:32.827+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:32.904 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:32.898+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:33.025 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:33.015+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:33.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:33.274+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:33.538 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:33.535+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:33.611 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:33.608+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:33.837 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:33.834+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:33.863 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:33.861+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:34.055 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:34.052+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:34.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:34.316+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:34.509 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:34.506+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:34.573 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:34.570+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:34.831 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:34.828+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:34.866 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:34.864+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:35.091 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:35.088+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:35.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:35.291+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:35.527 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:35.522+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:35.592 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:35.588+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:35.785 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:35.783+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:35.820 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:35.817+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:36.143 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:36.139+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:36.252 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:36.249+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:36.505 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:36.502+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:36.635 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:36.632+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:36.741 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:36.737+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:36.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:36.782+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:37.138 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:37.133+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:37.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:37.221+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:37.545 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:37.543+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:37.665 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:37.661+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:37.718 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:37.713+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:37.750 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:37.746+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:38.155 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:38.152+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:38.259 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:38.256+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:38.526 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:38.523+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:38.665 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:38.662+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:38.739 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:38.737+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:38.743 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:38.740+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:39.189 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:39.186+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:39.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:39.297+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:39.482 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:39.478+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:39.651 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:39.648+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:39.724 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:39.720+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:39.744 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:39.740+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:40.185 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:40.182+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:40.348 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:40.345+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:40.534 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:40.532+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:40.622 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:40.619+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:40.718 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:40.715+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:40.778 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:40.774+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:41.226 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:41.222+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:41.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:41.317+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:41.575 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:41.572+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:41.580 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:41.576+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:41.735 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:41.731+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:41.745 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:41.740+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:42.197 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:42.194+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:42.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:42.312+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:42.539 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:42.536+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:42.560 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:42.557+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:42.698 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:42.694+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:42.765 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:42.762+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:43.185 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:43.182+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:43.343 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:43.340+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:43.533 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:43.529+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:43.595 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:43.592+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:43.681 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:43.677+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:43.748 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:43.735+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:44.185 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:44.181+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:44.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:44.353+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:44.575 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:44.571+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:44.637 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:44.635+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:44.689 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:44.685+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:44.693 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:44.691+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:45.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:45.168+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:45.242 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:04:45.239+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 46sec e640 v640) 2019-12-12T00:04:45.350 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:45.346+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:45.592 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:45.589+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:45.683 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:45.679+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:45.683 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:45.677+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:45.709 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:45.706+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:46.165 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:46.162+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:46.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:46.317+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:46.620 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:46.617+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:46.706 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:46.704+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:46.712 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:46.710+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:46.715 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:46.713+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:47.187 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:47.186+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:47.375 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:47.372+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:47.661 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:47.658+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:47.665 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:47.662+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:47.760 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:47.758+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:47.765 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:47.763+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:48.210 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:48.207+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:48.378 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:48.376+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:48.690 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:48.687+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:48.709 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:48.706+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:48.725 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:48.721+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:48.726 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:48.723+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:49.166 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:49.163+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:49.167 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:49.166+0000 1694f700 -1 osd.4 640 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121522 2.fs0 2:f235c3be:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%eb%8c%8b%e0%a4%81%eb%8c%8c%e9%90%8b%ef%84%90%e2%98%83%e0%a4%81%eb%8c%8b%e2%98%83%e9%90%8c%e1%a0%83%e3%9c%85%e3%9c%86%e3%9c%84%e1%a0%81:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~132 in=132b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (58) in=78b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,setxattr user.rgw.x-amz-meta-36908a8026aa48dd9c398a0bcca27901 (46) in=98b,setxattr user.rgw.x-amz-meta-3db6eab266e14f3aa4a4a455455ec2ad (46) in=98b,setxattr user.rgw.x-amz-meta-3febe117522448a5aef72e2708ca35da (46) in=98b,setxattr user.rgw.x-amz-meta-75ae8db0ae2d435789f0f9c43d157a88 (46) in=98b,setxattr user.rgw.x-amz-meta-82b9e46f2f124c2bb492a1ad4a3befe9 (46) in=98b,setxattr user.rgw.x-amz-meta-8784b635587847e99810873833cd7134 (45) in=97b,setxattr user.rgw.x-amz-meta-88fcd4e409cc4878a86c9f47f10b1c72 (46) in=98b,setxattr user.rgw.x-amz-meta-cb1d8a7a1dfc41b8bb90abbfe6c0750f (45) in=97b,setxattr user.rgw.x-amz-meta-d410d18666934ecfb04c6c117ccfc702 (45) in=97b,setxattr user.rgw.x-amz-meta-e4e9aa0a9c0749c0b798ac8deded849e (45) in=97b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e640) 2019-12-12T00:04:49.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:04:49.403+0000 1694f700 -1 osd.7 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:29.448164+0000 front 2019-12-12T00:03:29.446809+0000 (oldest deadline 2019-12-12T00:04:10.838286+0000) 2019-12-12T00:04:49.675 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:04:49.668+0000 1694f700 -1 osd.6 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.011789+0000 front 2019-12-12T00:03:24.971778+0000 (oldest deadline 2019-12-12T00:04:10.074939+0000) 2019-12-12T00:04:49.681 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:04:49.678+0000 1694f700 -1 osd.2 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:27.751090+0000 front 2019-12-12T00:03:27.837214+0000 (oldest deadline 2019-12-12T00:04:10.639125+0000) 2019-12-12T00:04:49.725 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:04:49.722+0000 1694f700 -1 osd.1 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:28.582950+0000 front 2019-12-12T00:03:28.628149+0000 (oldest deadline 2019-12-12T00:04:12.653629+0000) 2019-12-12T00:04:49.750 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:49.745+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:50.149 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:50.146+0000 1694f700 -1 osd.4 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:30.104077+0000 front 2019-12-12T00:03:29.676147+0000 (oldest deadline 2019-12-12T00:04:15.564669+0000) 2019-12-12T00:04:50.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:50.148+0000 1694f700 -1 osd.4 640 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121522 2.fs0 2:f235c3be:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%eb%8c%8b%e0%a4%81%eb%8c%8c%e9%90%8b%ef%84%90%e2%98%83%e0%a4%81%eb%8c%8b%e2%98%83%e9%90%8c%e1%a0%83%e3%9c%85%e3%9c%86%e3%9c%84%e1%a0%81:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~132 in=132b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (58) in=78b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,setxattr user.rgw.x-amz-meta-36908a8026aa48dd9c398a0bcca27901 (46) in=98b,setxattr user.rgw.x-amz-meta-3db6eab266e14f3aa4a4a455455ec2ad (46) in=98b,setxattr user.rgw.x-amz-meta-3febe117522448a5aef72e2708ca35da (46) in=98b,setxattr user.rgw.x-amz-meta-75ae8db0ae2d435789f0f9c43d157a88 (46) in=98b,setxattr user.rgw.x-amz-meta-82b9e46f2f124c2bb492a1ad4a3befe9 (46) in=98b,setxattr user.rgw.x-amz-meta-8784b635587847e99810873833cd7134 (45) in=97b,setxattr user.rgw.x-amz-meta-88fcd4e409cc4878a86c9f47f10b1c72 (46) in=98b,setxattr user.rgw.x-amz-meta-cb1d8a7a1dfc41b8bb90abbfe6c0750f (45) in=97b,setxattr user.rgw.x-amz-meta-d410d18666934ecfb04c6c117ccfc702 (45) in=97b,setxattr user.rgw.x-amz-meta-e4e9aa0a9c0749c0b798ac8deded849e (45) in=97b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e640) 2019-12-12T00:04:50.293 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:04:50.288+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 46sec e640 v640) 2019-12-12T00:04:50.737 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:04:50.735+0000 1694f700 -1 osd.3 640 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:03:25.037804+0000 front 2019-12-12T00:03:25.036451+0000 (oldest deadline 2019-12-12T00:04:10.312218+0000) 2019-12-12T00:04:51.351 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:51.325+0000 1694f700 -1 osd.4 641 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121522 2.fs0 2:f235c3be:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%eb%8c%8b%e0%a4%81%eb%8c%8c%e9%90%8b%ef%84%90%e2%98%83%e0%a4%81%eb%8c%8b%e2%98%83%e9%90%8c%e1%a0%83%e3%9c%85%e3%9c%86%e3%9c%84%e1%a0%81:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~132 in=132b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (58) in=78b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,setxattr user.rgw.x-amz-meta-36908a8026aa48dd9c398a0bcca27901 (46) in=98b,setxattr user.rgw.x-amz-meta-3db6eab266e14f3aa4a4a455455ec2ad (46) in=98b,setxattr user.rgw.x-amz-meta-3febe117522448a5aef72e2708ca35da (46) in=98b,setxattr user.rgw.x-amz-meta-75ae8db0ae2d435789f0f9c43d157a88 (46) in=98b,setxattr user.rgw.x-amz-meta-82b9e46f2f124c2bb492a1ad4a3befe9 (46) in=98b,setxattr user.rgw.x-amz-meta-8784b635587847e99810873833cd7134 (45) in=97b,setxattr user.rgw.x-amz-meta-88fcd4e409cc4878a86c9f47f10b1c72 (46) in=98b,setxattr user.rgw.x-amz-meta-cb1d8a7a1dfc41b8bb90abbfe6c0750f (45) in=97b,setxattr user.rgw.x-amz-meta-d410d18666934ecfb04c6c117ccfc702 (45) in=97b,setxattr user.rgw.x-amz-meta-e4e9aa0a9c0749c0b798ac8deded849e (45) in=97b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e640) 2019-12-12T00:04:52.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:52.319+0000 1694f700 -1 osd.4 642 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121522 2.fs0 2:f235c3be:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%eb%8c%8b%e0%a4%81%eb%8c%8c%e9%90%8b%ef%84%90%e2%98%83%e0%a4%81%eb%8c%8b%e2%98%83%e9%90%8c%e1%a0%83%e3%9c%85%e3%9c%86%e3%9c%84%e1%a0%81:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~132 in=132b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (58) in=78b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,setxattr user.rgw.x-amz-meta-36908a8026aa48dd9c398a0bcca27901 (46) in=98b,setxattr user.rgw.x-amz-meta-3db6eab266e14f3aa4a4a455455ec2ad (46) in=98b,setxattr user.rgw.x-amz-meta-3febe117522448a5aef72e2708ca35da (46) in=98b,setxattr user.rgw.x-amz-meta-75ae8db0ae2d435789f0f9c43d157a88 (46) in=98b,setxattr user.rgw.x-amz-meta-82b9e46f2f124c2bb492a1ad4a3befe9 (46) in=98b,setxattr user.rgw.x-amz-meta-8784b635587847e99810873833cd7134 (45) in=97b,setxattr user.rgw.x-amz-meta-88fcd4e409cc4878a86c9f47f10b1c72 (46) in=98b,setxattr user.rgw.x-amz-meta-cb1d8a7a1dfc41b8bb90abbfe6c0750f (45) in=97b,setxattr user.rgw.x-amz-meta-d410d18666934ecfb04c6c117ccfc702 (45) in=97b,setxattr user.rgw.x-amz-meta-e4e9aa0a9c0749c0b798ac8deded849e (45) in=97b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e640) 2019-12-12T00:04:53.283 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:04:53.283+0000 1694f700 -1 osd.4 642 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:121522 2.fs0 2:f235c3be:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%eb%8c%8b%e0%a4%81%eb%8c%8c%e9%90%8b%ef%84%90%e2%98%83%e0%a4%81%eb%8c%8b%e2%98%83%e9%90%8c%e1%a0%83%e3%9c%85%e3%9c%86%e3%9c%84%e1%a0%81:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~132 in=132b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (58) in=78b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,setxattr user.rgw.x-amz-meta-36908a8026aa48dd9c398a0bcca27901 (46) in=98b,setxattr user.rgw.x-amz-meta-3db6eab266e14f3aa4a4a455455ec2ad (46) in=98b,setxattr user.rgw.x-amz-meta-3febe117522448a5aef72e2708ca35da (46) in=98b,setxattr user.rgw.x-amz-meta-75ae8db0ae2d435789f0f9c43d157a88 (46) in=98b,setxattr user.rgw.x-amz-meta-82b9e46f2f124c2bb492a1ad4a3befe9 (46) in=98b,setxattr user.rgw.x-amz-meta-8784b635587847e99810873833cd7134 (45) in=97b,setxattr user.rgw.x-amz-meta-88fcd4e409cc4878a86c9f47f10b1c72 (46) in=98b,setxattr user.rgw.x-amz-meta-cb1d8a7a1dfc41b8bb90abbfe6c0750f (45) in=97b,setxattr user.rgw.x-amz-meta-d410d18666934ecfb04c6c117ccfc702 (45) in=97b,setxattr user.rgw.x-amz-meta-e4e9aa0a9c0749c0b798ac8deded849e (45) in=97b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e640) 2019-12-12T00:05:15.472 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:33:02.406 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.498 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:33:02.432 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.512 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:33:02.445 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.534 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:33:02.467 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.552 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:33:02.458 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.555 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:33:02.461 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.617 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:33:02.551 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.658 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:33:02.592 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:15.695 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:33:02.601 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:05:23.147 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:23.056+0000 36790700 -1 osd.0 646 build_incremental_map_msg missing incremental map 646 2019-12-12T00:05:23.166 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:23.102+0000 31f87700 -1 osd.0 646 build_incremental_map_msg missing incremental map 646 2019-12-12T00:05:23.497 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:23.472+0000 35f8f700 -1 osd.0 646 build_incremental_map_msg missing incremental map 646 2019-12-12T00:05:23.755 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:23.727+0000 36790700 -1 osd.0 646 build_incremental_map_msg missing incremental map 646 2019-12-12T00:05:25.198 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:25.185+0000 34f8d700 -1 osd.0 646 build_incremental_map_msg missing incremental map 646 2019-12-12T00:05:25.210 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:25.204+0000 33f8b700 -1 osd.0 646 build_incremental_map_msg missing incremental map 646 2019-12-12T00:05:49.211 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:49.210+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:49.577 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:49.574+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:50.209 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:50.208+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:50.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:50.552+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:51.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:51.109+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:51.237 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:51.235+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:51.396 INFO:teuthology.orchestra.run.smithi079.stderr:testMetadataOnPut (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:05:51.476 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:51.475+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:51.521 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:51.518+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:52.129 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:52.126+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:52.283 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:52.281+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:52.347 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:52.346+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:52.512 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:52.509+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:52.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:52.585+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:53.106 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:53.104+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:53.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:53.334+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:53.390 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:53.388+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:53.517 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:53.515+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:53.574 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:53.572+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:54.092 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:54.089+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:54.303 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:54.303+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:54.411 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:54.409+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:54.541 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:54.537+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:54.580 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:54.579+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:55.124 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:55.122+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:55.262 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:55.261+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:55.353 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:55.351+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:05:55.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:55.375+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:55.503 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:55.501+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:55.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:55.606+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:56.177 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:56.174+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:56.258 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:56.256+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:56.385 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:56.383+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:05:56.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:56.423+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:56.473 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:56.470+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:56.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:56.622+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:57.143 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:57.140+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:57.217 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:57.215+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:57.351 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:57.350+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:05:57.446 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:57.442+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:57.478 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:57.476+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:57.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:57.649+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:58.194 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:58.192+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:58.219 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:58.216+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:58.342 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:58.340+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:05:58.343 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:58.341+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:05:58.436 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:58.432+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:58.469 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:58.466+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:58.697 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:58.688+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:05:58.998 INFO:teuthology.orchestra.run.smithi079.stderr:testNameLimit (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:05:59.067 INFO:teuthology.orchestra.run.smithi079.stderr:testNoContentLengthForPut (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:05:59.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:05:59.199+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:05:59.238 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:05:59.236+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:05:59.371 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:59.370+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:05:59.373 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:05:59.373+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:05:59.450 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:05:59.448+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:05:59.502 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:05:59.500+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:05:59.571 INFO:teuthology.orchestra.run.smithi079.stderr:testPostOnFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:05:59.681 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:05:59.678+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:00.232 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:00.230+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:00.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:00.280+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:00.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:00.400+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:00.408 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:00.401+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:00.454 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:00.451+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:00.461 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:00.458+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:00.696 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:00.693+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:01.189 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:01.187+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:01.270 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:01.267+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:01.404 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:01.402+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:01.405 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:01.403+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:01.470 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:01.466+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:01.476 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:01.474+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:01.682 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:01.681+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:02.178 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:02.175+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:02.312 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:02.310+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:02.431 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:02.428+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:02.432 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:02.430+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:02.444 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:02.442+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:02.473 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:02.464+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:02.657 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:02.655+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:03.202 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:03.184+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:03.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:03.319+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:03.434 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:03.430+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:03.442 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:03.439+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:03.448 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:03.445+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:03.449 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:03.447+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:03.689 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:03.687+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:04.187 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:04.185+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:04.323 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:04.320+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:04.441 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:04.438+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:04.443 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:04.440+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:04.449 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:04.446+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:04.450 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:04.447+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:04.670 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:04.668+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:05.151 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:05.149+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:05.340 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:05.332+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:05.395 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:05.392+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:05.395 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:05.393+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:05.457 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:05.454+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:05.461 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:05.459+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:05.715 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:05.712+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:06.116 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:06.112+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:06.302 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:06.298+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:06.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:06.396+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:06.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:06.397+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:06.474 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:06.471+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:06.488 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:06.486+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:06.685 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:06.682+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:07.143 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:07.138+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:07.308 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:07.305+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:07.358 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:07.355+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:07.359 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:07.356+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:07.491 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:07.488+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:07.499 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:07.497+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:07.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:07.656+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:08.102 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:08.100+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:08.271 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:08.268+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:08.311 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:08.308+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:08.312 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:08.309+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:08.455 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:08.451+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:08.509 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:08.506+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:08.662 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:08.654+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:09.084 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:09.081+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:09.272 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:09.269+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:09.289 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:09.285+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:09.289 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:09.287+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:09.466 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:09.462+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:09.552 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:09.550+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:09.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:09.640+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:10.088 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:10.087+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:10.242 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:10.239+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:10.243 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:10.241+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:10.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:10.320+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:10.456 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:10.452+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:10.548 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:10.545+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:10.684 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:10.681+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:11.123 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:11.120+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:11.273 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:11.270+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:11.273 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:11.271+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:11.360 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:11.358+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:11.463 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:11.461+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:11.561 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:11.555+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:11.716 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:11.713+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:12.093 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:12.090+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:12.301 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:12.298+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:12.302 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:12.300+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:12.378 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:12.377+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:12.509 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:12.508+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:12.570 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:12.568+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:12.740 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:12.737+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:13.053 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:13.051+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:13.331 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:13.330+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:13.333 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:13.332+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:13.416 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:13.414+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:13.511 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:13.509+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:13.620 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:13.618+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:13.704 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:13.702+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:14.049 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:14.044+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:14.300 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:14.293+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:14.300 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:14.295+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:14.382 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:14.379+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:14.556 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:14.554+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:14.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:14.633+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:14.698 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:14.695+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:15.003 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:15.001+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:15.279 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:15.276+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:15.280 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:15.278+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:15.433 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:15.430+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:15.567 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:15.564+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:15.616 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:15.614+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:15.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:15.740+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:16.021 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:16.018+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:16.264 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:16.262+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:16.265 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:16.263+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:16.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:16.460+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:16.584 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:16.582+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:16.593 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:16.590+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:16.777 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:16.774+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:17.008 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:17.006+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:17.268 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:17.267+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:17.269 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:17.267+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:17.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:17.418+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:17.560 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:17.558+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:17.581 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:17.578+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:17.801 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:17.791+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:18.003 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:18.000+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:18.226 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:18.223+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:18.226 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:18.224+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:18.435 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:18.433+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:18.582 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:18.580+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:18.584 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:18.581+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:18.796 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:18.794+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:19.048 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:19.046+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:19.231 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:19.228+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:19.231 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:19.229+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:19.424 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:19.422+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:19.585 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:19.583+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:19.643 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:19.641+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:19.813 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:19.811+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:20.065 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:20.057+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:20.210 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:20.207+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:20.211 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:20.209+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:20.421 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:20.419+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:20.567 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:20.565+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:20.638 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:06:20.634+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e646 v646) 2019-12-12T00:06:20.677 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:20.675+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:20.845 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:20.842+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:21.051 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:21.049+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:21.182 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:21.178+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:21.182 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:21.181+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:21.391 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:21.388+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:21.532 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:21.529+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:21.674 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:21.673+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:21.853 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:21.850+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:22.100 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:22.097+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:22.204 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:22.201+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:22.205 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:22.203+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:22.359 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:22.357+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:22.570 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:22.568+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:22.641 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:22.638+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:22.868 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:22.866+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:23.112 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:23.110+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:23.167 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:23.164+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:23.168 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:23.166+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:23.357 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:23.355+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:23.552 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:23.548+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:23.676 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:23.672+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:23.913 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:23.911+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:24.085 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:24.082+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:24.208 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:24.206+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:24.211 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:24.210+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:24.409 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:24.407+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:24.564 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:24.562+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:24.703 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:24.700+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:24.874 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:24.871+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:25.080 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:25.079+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:25.173 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:25.171+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:25.174 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:25.172+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:25.457 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:25.455+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:25.525 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:25.522+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:25.651 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:06:25.649+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e646 v646) 2019-12-12T00:06:25.711 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:25.709+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:25.849 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:25.845+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:26.046 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:26.044+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:26.220 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:26.218+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:26.220 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:26.219+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:26.458 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:26.454+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:26.530 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:26.526+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:26.729 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:26.728+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:26.866 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:26.864+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:27.101 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:27.099+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:27.248 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:27.238+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:27.248 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:27.240+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:27.415 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:27.413+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:27.532 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:27.529+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:27.720 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:27.719+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:27.883 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:27.881+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:28.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:28.154+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:28.293 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:28.290+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:28.296 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:28.292+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:28.446 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:28.445+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:28.490 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:28.488+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:28.714 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:28.708+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:28.918 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:28.915+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:29.159 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:29.157+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:29.315 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:29.313+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:29.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:29.316+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:29.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:29.462+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:29.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:29.466+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:29.730 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:29.727+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:29.908 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:29.906+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:30.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:30.198+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:30.339 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:30.337+0000 1694f700 -1 osd.0 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:05:17.752810+0000 (oldest deadline 2019-12-12T00:05:57.752810+0000) 2019-12-12T00:06:30.340 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:30.339+0000 1694f700 -1 osd.0 646 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:30.469 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:06:30.465+0000 1694f700 -1 osd.2 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.440221+0000 front 2019-12-12T00:05:06.319003+0000 (oldest deadline 2019-12-12T00:05:50.793115+0000) 2019-12-12T00:06:30.497 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:06:30.494+0000 1694f700 -1 osd.6 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.448715+0000 front 2019-12-12T00:05:06.458116+0000 (oldest deadline 2019-12-12T00:05:51.503926+0000) 2019-12-12T00:06:30.687 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:06:30.684+0000 1694f700 -1 osd.1 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.780451+0000 front 2019-12-12T00:05:06.872992+0000 (oldest deadline 2019-12-12T00:05:50.779396+0000) 2019-12-12T00:06:30.718 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:06:30.714+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e646 v646) 2019-12-12T00:06:30.902 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:06:30.900+0000 1694f700 -1 osd.7 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.267937+0000 front 2019-12-12T00:05:05.983757+0000 (oldest deadline 2019-12-12T00:05:49.426962+0000) 2019-12-12T00:06:31.209 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:06:31.208+0000 1694f700 -1 osd.4 646 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:05:06.859619+0000 front 2019-12-12T00:05:06.912749+0000 (oldest deadline 2019-12-12T00:05:48.533798+0000) 2019-12-12T00:06:31.555 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:06:31.469+0000 1694f700 -1 osd.0 647 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e644) 2019-12-12T00:06:56.237 INFO:teuthology.orchestra.run.smithi079.stderr:testRangedGets (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:07:07.142 INFO:teuthology.orchestra.run.smithi079.stderr:testSerialization (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:07:09.652 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:07:09.632+0000 36790700 -1 osd.4 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:09.703 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:07:09.640+0000 36790700 -1 osd.4 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:09.770 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:07:09.558+0000 32788700 -1 osd.0 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:09.841 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:07:09.579+0000 32788700 -1 osd.0 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:09.847 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:07:09.597+0000 36790700 -1 osd.0 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:09.873 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:07:09.694+0000 31f87700 -1 osd.0 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:09.883 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:07:09.691+0000 32788700 -1 osd.4 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:09.887 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:07:09.706+0000 3378a700 -1 osd.4 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:10.091 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:07:09.901+0000 38794700 -1 osd.4 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:10.131 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:07:09.987+0000 36f91700 -1 osd.0 650 build_incremental_map_msg missing incremental map 650 2019-12-12T00:07:10.429 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:34:57.362 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.442 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:34:57.375 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.446 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:34:57.352 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.451 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:34:57.357 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.453 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:34:57.386 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.455 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:34:57.388 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.614 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:34:57.520 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.636 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:34:57.570 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:10.679 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:34:57.612 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:07:19.940 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:07:19.931+0000 36f91700 -1 osd.3 652 build_incremental_map_msg missing incremental map 651 2019-12-12T00:08:14.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:14.192+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:14.604 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:14.601+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:14.769 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:14.766+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:15.233 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:15.231+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:15.626 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:15.625+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:15.630 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:15.628+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:15.758 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:15.754+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:16.274 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:16.271+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:16.647 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:16.644+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:16.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:16.651+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:16.758 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:16.755+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:17.059 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:17.056+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:17.255 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:17.253+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:17.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:17.600+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:17.665 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:17.661+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:17.787 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:17.785+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:18.064 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:18.056+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:18.305 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:18.295+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:18.603 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:18.595+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:18.636 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:18.629+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:18.759 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:18.756+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:19.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:19.069+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:19.278 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:19.275+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:19.628 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:19.625+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:19.629 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:19.625+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:19.756 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:19.752+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:20.036 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:20.032+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:20.268 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:20.264+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:20.596 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:20.593+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:20.642 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:20.639+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:20.801 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:20.798+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:21.023 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:21.019+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:21.276 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:21.273+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:21.613 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:21.610+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:21.694 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:21.690+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:21.802 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:21.799+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:21.989 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:21.986+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:22.311 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:22.308+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:22.569 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:22.567+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:22.698 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:22.695+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:22.770 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:22.767+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:22.984 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:22.982+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:23.327 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:23.324+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:23.566 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:23.563+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:23.750 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:23.747+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:23.806 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:23.804+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:23.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:23.942+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:24.375 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:24.372+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:24.539 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:24.536+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:24.722 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:24.719+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:24.780 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:24.777+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:24.928 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:24.925+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:25.368 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:25.362+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:25.548 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:25.546+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:25.753 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:25.750+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:25.753 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:25.751+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:25.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:25.967+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:26.398 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:26.394+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:26.565 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:26.562+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:26.799 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:26.795+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:26.802 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:26.799+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:26.950 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:26.948+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:27.354 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:27.350+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:27.591 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:27.590+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:27.785 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:27.783+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:27.833 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:27.830+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:27.975 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:27.971+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:28.381 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:28.378+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:28.628 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:28.625+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:28.832 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:28.829+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:28.869 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:28.865+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:28.931 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:28.928+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:29.431 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:29.428+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:29.649 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:29.648+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:29.817 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:29.814+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:29.864 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:29.861+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:29.892 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:29.890+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:30.425 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:30.423+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:30.626 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:30.624+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:30.864 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:30.862+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:30.867 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:30.864+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:30.934 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:30.932+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:31.458 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:31.456+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:31.599 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:31.596+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:31.819 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:31.817+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:31.884 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:31.881+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:31.981 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:31.978+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:32.469 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:32.466+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:32.631 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:32.628+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:32.801 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:32.799+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:32.879 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:32.877+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:32.975 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:32.973+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:33.466 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:33.463+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:33.620 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:33.617+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:33.844 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:33.842+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:33.878 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:33.875+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:34.021 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:34.014+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:34.486 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:34.483+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:34.586 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:34.583+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:34.850 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:34.847+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:34.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:34.899+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:34.972 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:34.970+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:35.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:35.449+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:35.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:35.606+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:35.873 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:35.871+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:35.941 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:35.938+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:35.943 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:35.940+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:36.248 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:36.245+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:36.465 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:36.458+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:36.657 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:36.647+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:36.828 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:36.826+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:36.927 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:36.924+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:36.979 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:36.975+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:37.269 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:37.266+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:37.477 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:37.474+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:37.663 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:37.659+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:37.817 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:37.812+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:37.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:37.900+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:38.001 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:37.997+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:38.317 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:38.315+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:38.510 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:38.508+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:38.623 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:38.619+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:38.783 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:38.780+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:38.922 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:38.920+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:39.046 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:39.043+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:39.348 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:39.347+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:39.524 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:39.523+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:39.589 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:39.587+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:39.743 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:39.741+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:39.911 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:39.907+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:40.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:40.003+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:40.370 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:40.368+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:40.567 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:40.565+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:40.594 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:40.589+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:40.738 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:40.735+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:40.951 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:40.948+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:41.012 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:41.011+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:41.409 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:41.407+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:41.554 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:41.552+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:41.588 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:41.587+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:41.755 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:41.751+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:41.982 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:41.978+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:42.012 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:42.003+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:42.403 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:42.402+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:42.545 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:42.542+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:42.568 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:42.566+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:42.774 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:42.771+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:42.985 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:42.984+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:43.048 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:43.045+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:43.426 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:43.423+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:43.585 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:43.579+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:43.619 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:43.617+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:43.825 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:43.823+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:43.952 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:43.950+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:44.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:44.068+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:44.388 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:44.386+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:44.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:44.605+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:44.665 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:44.663+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:44.859 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:44.857+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:44.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:44.941+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:45.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:45.115+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:45.368 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:45.366+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:45.577 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:45.574+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:45.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:45.632+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:45.900 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:45.897+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:45.917 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:45.914+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:46.148 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:46.146+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:46.203 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:08:46.200+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 43sec e652 v652) 2019-12-12T00:08:46.342 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:46.338+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:46.568 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:46.566+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:46.680 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:46.678+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:46.902 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:46.900+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:46.942 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:46.938+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:47.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:47.174+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:47.349 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:47.346+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:47.572 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:47.570+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:47.666 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:47.663+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:47.882 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:47.879+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:47.922 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:47.920+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:48.140 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:48.138+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:48.303 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:48.301+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:48.576 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:48.572+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:48.695 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:48.692+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:48.839 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:48.836+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:48.947 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:48.945+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:49.139 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:49.136+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:49.269 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:49.255+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:49.565 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:49.552+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:49.720 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:49.711+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:49.797 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:49.790+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:49.910 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:49.909+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:50.170 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:50.167+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:50.306 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:50.304+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:50.612 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:50.606+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:08:50.685 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:08:50.682+0000 1694f700 -1 osd.6 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.970702+0000 front 2019-12-12T00:07:32.223602+0000 (oldest deadline 2019-12-12T00:08:14.185902+0000) 2019-12-12T00:08:50.761 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:08:50.756+0000 1694f700 -1 osd.3 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:32.592860+0000 front 2019-12-12T00:07:32.613448+0000 (oldest deadline 2019-12-12T00:08:14.274827+0000) 2019-12-12T00:08:50.865 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:08:50.863+0000 1694f700 -1 osd.1 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:29.566904+0000 front 2019-12-12T00:07:29.660322+0000 (oldest deadline 2019-12-12T00:08:14.057158+0000) 2019-12-12T00:08:51.165 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:08:51.163+0000 1694f700 -1 osd.7 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.814810+0000 front 2019-12-12T00:07:31.718252+0000 (oldest deadline 2019-12-12T00:08:16.307912+0000) 2019-12-12T00:08:51.248 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:08:51.246+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 43sec e652 v652) 2019-12-12T00:08:51.342 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:08:51.333+0000 1694f700 -1 osd.4 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.493102+0000 front 2019-12-12T00:07:31.364267+0000 (oldest deadline 2019-12-12T00:08:35.280645+0000) 2019-12-12T00:08:51.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:08:51.606+0000 1694f700 -1 osd.2 652 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:07:31.245651+0000 front 2019-12-12T00:07:31.246628+0000 (oldest deadline 2019-12-12T00:08:15.132282+0000) 2019-12-12T00:09:23.057 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:09:22.734+0000 32f89700 -1 osd.3 656 build_incremental_map_msg missing incremental map 656 2019-12-12T00:09:23.134 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:09:22.772+0000 33f8b700 -1 osd.3 656 build_incremental_map_msg missing incremental map 656 2019-12-12T00:09:23.163 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:09:22.881+0000 3378a700 -1 osd.3 656 build_incremental_map_msg missing incremental map 656 2019-12-12T00:09:23.210 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:09:22.967+0000 36790700 -1 osd.3 656 build_incremental_map_msg missing incremental map 656 2019-12-12T00:09:23.345 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:09:23.024+0000 35f8f700 -1 osd.3 656 build_incremental_map_msg missing incremental map 656 2019-12-12T00:09:23.389 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:09:23.042+0000 3478c700 -1 osd.3 656 build_incremental_map_msg missing incremental map 656 2019-12-12T00:09:25.055 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:24.965+0000 31786700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:25.129 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:24.979+0000 31786700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:25.150 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:24.982+0000 31786700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:25.169 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:25.004+0000 32788700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:25.292 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:25.159+0000 32f89700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:25.312 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:25.162+0000 32f89700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:27.967 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:27.930+0000 32f89700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:30.131 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:30.129+0000 34f8d700 -1 osd.0 658 build_incremental_map_msg missing incremental map 646 2019-12-12T00:09:30.131 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:30.129+0000 34f8d700 -1 osd.0 658 build_incremental_map_msg missing incremental map 650 2019-12-12T00:09:30.136 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:30.135+0000 34f8d700 -1 osd.0 658 build_incremental_map_msg missing incremental map 658 2019-12-12T00:09:49.925 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:49.914+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:49.925 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:49.921+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:50.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:50.933+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:50.941 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:50.938+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:51.431 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:51.429+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:51.756 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:51.754+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:51.928 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:51.919+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:51.928 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:51.923+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:52.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:52.296+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:52.482 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:52.479+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:52.746 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:52.744+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:52.884 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:52.882+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:52.979 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:52.977+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:53.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:53.292+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:53.477 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:53.474+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:53.795 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:53.793+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:53.928 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:53.927+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:54.004 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:54.001+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:54.312 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:54.310+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:54.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:54.328+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:09:54.472 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:54.470+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:54.788 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:54.786+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:54.953 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:54.951+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:54.999 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:54.996+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:55.335 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:55.333+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:09:55.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:55.342+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:55.517 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:55.515+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:55.807 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:55.803+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:55.912 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:55.910+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:56.015 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:56.012+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:56.320 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:56.318+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:09:56.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:56.391+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:56.500 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:56.499+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:56.822 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:56.820+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:56.944 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:56.942+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:56.974 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:56.972+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:57.356 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:57.355+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:09:57.362 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:57.360+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:57.501 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:57.500+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:57.777 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:57.774+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:57.967 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:57.965+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:57.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:57.992+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:58.317 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:58.316+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:09:58.353 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:58.347+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:58.461 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:58.459+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:58.793 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:58.789+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:59.006 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:59.005+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:09:59.018 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:09:59.016+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:09:59.304 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:09:59.302+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:09:59.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:09:59.324+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:09:59.434 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:09:59.431+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:09:59.802 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:09:59.799+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:09:59.982 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:09:59.980+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:00.059 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:00.057+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:00.271 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:00.269+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:00.271 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:00.270+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:00.291 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:00.289+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:00.466 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:00.464+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:00.825 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:00.823+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:00.946 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:00.944+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:01.105 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:01.104+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:01.286 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:01.283+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:01.288 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:01.287+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:01.305 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:01.303+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:01.440 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:01.437+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:01.820 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:01.817+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:01.962 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:01.960+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:02.093 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:02.091+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:02.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:02.269+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:02.291 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:02.287+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:02.292 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:02.288+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:02.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:02.394+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:02.874 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:02.870+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:02.959 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:02.957+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:03.102 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:03.100+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:03.254 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:03.252+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:03.263 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:03.261+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:03.267 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:03.266+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:03.428 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:03.425+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:03.852 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:03.849+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:03.920 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:03.918+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:04.081 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:04.079+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:04.290 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:04.286+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:04.290 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:04.289+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:04.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:04.300+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:04.401 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:04.399+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:04.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:04.902+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:04.907 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:04.904+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:05.118 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:05.116+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:05.285 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:05.283+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:05.308 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:05.305+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:05.308 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:05.307+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:05.439 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:05.436+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:05.883 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:05.881+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:05.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:05.931+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:06.139 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:06.136+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:06.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:06.242+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:06.318 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:06.313+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:06.319 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:06.317+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:06.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:06.490+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:06.855 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:06.852+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:06.940 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:06.938+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:07.131 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:07.128+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:07.261 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:07.258+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:07.324 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:07.322+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:07.326 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:07.325+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:07.461 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:07.460+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:07.635 INFO:teuthology.orchestra.run.smithi079.stderr:testStackedOverwrite (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:10:07.748 INFO:teuthology.orchestra.run.smithi079.stderr:testTooLongName (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:10:07.898 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:07.893+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:07.929 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:07.927+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:08.143 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:08.140+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:08.302 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:08.301+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:08.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:08.370+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:08.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:08.372+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:08.454 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:08.452+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:08.948 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:08.945+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:08.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:08.971+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:09.106 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:09.105+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:09.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:09.263+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:09.354 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:09.351+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:09.355 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:09.353+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:09.485 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:09.483+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:09.983 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:09.981+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:09.997 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:09.994+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:10.157 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:10.154+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:10.259 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:10.257+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:10.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:10.386+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:10.391 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:10.388+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:10.447 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:10.444+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:10.960 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:10.957+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:11.024 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:11.021+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:11.115 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:11.112+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:11.290 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:11.288+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:11.424 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:11.421+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:11.433 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:11.430+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:11.434 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:11.432+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:12.006 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:12.004+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:12.010 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:12.008+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:12.142 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:12.141+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:12.253 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:12.250+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:12.438 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:12.436+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:12.478 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:12.476+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:12.479 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:12.479+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:12.984 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:12.983+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:13.013 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:13.011+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:13.118 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:13.116+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:13.236 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:13.232+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:13.392 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:13.390+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:13.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:13.507+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:13.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:13.508+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:13.975 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:13.973+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:13.998 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:13.996+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:14.108 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:14.106+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:14.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:14.264+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:14.371 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:14.369+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:14.489 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:14.487+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:14.492 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:14.490+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:14.970 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:14.968+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:15.007 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:15.005+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:15.075 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:15.072+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:15.242 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:15.240+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:15.386 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:15.384+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:15.460 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:15.457+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:15.462 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:15.461+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:15.966 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:15.963+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:16.055 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:16.053+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:16.104 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:16.101+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:16.205 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:16.202+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:16.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:16.358+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:16.453 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:16.449+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:16.456 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:16.451+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:16.955 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:16.952+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:17.011 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:17.008+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:17.091 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:17.088+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:17.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:17.216+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:17.343 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:17.341+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:17.431 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:17.427+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:17.431 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:17.429+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:17.969 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:17.968+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:17.970 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:17.969+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:18.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:18.108+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:18.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:18.216+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:18.327 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:18.324+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:18.464 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:18.460+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:18.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:18.463+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:18.944 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:18.942+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:18.966 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:18.964+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:19.091 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:19.085+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:19.270 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:19.269+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:19.323 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:19.320+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:19.468 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:19.466+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:19.469 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:19.467+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:19.930 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:19.928+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:19.969 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:19.966+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:20.086 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:20.084+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:20.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:20.278+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:20.359 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:20.356+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:20.501 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:20.498+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:20.502 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:20.500+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:20.897 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:20.894+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:21.013 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:21.011+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:21.110 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:21.108+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:21.286 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:21.278+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:21.377 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:21.374+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:21.480 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:10:21.475+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e658 v658) 2019-12-12T00:10:21.508 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:21.500+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:21.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:21.501+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:21.915 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:21.913+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:22.030 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:22.028+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:22.162 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:22.160+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:22.255 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:22.253+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:22.400 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:22.398+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:22.508 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:22.505+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:22.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:22.507+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:22.960 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:22.955+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:23.044 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:23.041+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:23.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:23.156+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:23.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:23.282+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:23.380 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:23.378+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:23.483 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:23.481+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:23.484 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:23.483+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:23.920 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:23.918+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:24.016 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:24.014+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:24.188 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:24.175+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:24.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:24.334+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:24.363 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:24.361+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:24.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:24.452+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:24.455 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:24.454+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:24.918 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:24.916+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:25.029 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:25.026+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:25.188 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:25.184+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:25.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:25.306+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:25.390 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:25.388+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:25.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:25.419+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:25.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:25.420+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:25.940 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:25.938+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:26.031 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:26.028+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:26.240 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:26.238+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:26.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:26.274+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:26.413 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:26.411+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:26.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:26.412+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:26.439 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:26.436+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:26.495 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:10:26.490+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e658 v658) 2019-12-12T00:10:26.936 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:26.934+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:27.007 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:27.005+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:27.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:27.256+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:27.277 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:27.275+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:27.420 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:27.418+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:27.421 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:27.418+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:27.480 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:27.477+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:27.945 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:27.940+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:28.002 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:27.999+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:28.261 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:28.259+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:28.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:28.296+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:28.388 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:28.383+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:28.388 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:28.384+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:28.517 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:28.515+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:28.976 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:28.974+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:28.981 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:28.977+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:29.296 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:29.294+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:29.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:29.297+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:29.379 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:29.376+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:29.380 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:29.377+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:29.496 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:29.493+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:29.937 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:29.934+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:29.958 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:29.955+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:30.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:30.307+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:30.329 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:30.327+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:30.390 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:30.387+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:30.391 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:30.388+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:30.487 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:30.483+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:30.915 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:30.912+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:30.952 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:30.949+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:31.308 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:31.305+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:31.316 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:31.314+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:31.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:31.402+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:31.406 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:31.403+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:31.501 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:31.497+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:31.513 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:10:31.508+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e658 v658) 2019-12-12T00:10:31.909 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:31.907+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:31.930 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:31.927+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:32.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:32.318+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:32.337 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:32.332+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:32.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:32.411+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:32.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:32.411+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:32.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:32.465+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:32.904 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:32.901+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:32.920 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:32.917+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:33.322 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:33.320+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:33.353 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:33.352+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:33.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:33.397+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:33.403 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:33.400+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:33.423 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:33.420+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:33.872 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:33.871+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:33.905 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:33.902+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:34.341 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:34.340+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:34.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:34.340+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:34.381 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:34.379+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:34.402 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:34.399+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:34.405 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:34.403+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:34.906 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:34.905+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:34.936 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:34.934+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:35.301 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:35.299+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:35.307 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:35.305+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:35.367 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:35.364+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:35.374 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:35.368+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:35.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:35.405+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:35.898 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:10:35.896+0000 1694f700 -1 osd.6 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:08.225087+0000 front 2019-12-12T00:09:08.227254+0000 (oldest deadline 2019-12-12T00:09:51.645101+0000) 2019-12-12T00:10:35.961 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:10:35.958+0000 1694f700 -1 osd.1 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:04.486303+0000 front 2019-12-12T00:09:04.499389+0000 (oldest deadline 2019-12-12T00:09:49.433710+0000) 2019-12-12T00:10:36.317 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:10:36.315+0000 1694f700 -1 osd.4 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.254245+0000 front 2019-12-12T00:09:06.327894+0000 (oldest deadline 2019-12-12T00:09:49.739518+0000) 2019-12-12T00:10:36.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:10:36.323+0000 1694f700 -1 osd.7 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.447286+0000 front 2019-12-12T00:09:06.569396+0000 (oldest deadline 2019-12-12T00:09:51.718080+0000) 2019-12-12T00:10:36.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:36.325+0000 1694f700 -1 osd.0 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:09:19.870246+0000 (oldest deadline 2019-12-12T00:09:59.870246+0000) 2019-12-12T00:10:36.329 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:36.326+0000 1694f700 -1 osd.0 658 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e656) 2019-12-12T00:10:36.434 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:10:36.430+0000 1694f700 -1 osd.2 658 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:09:06.480299+0000 front 2019-12-12T00:09:06.381423+0000 (oldest deadline 2019-12-12T00:09:50.452894+0000) 2019-12-12T00:10:36.591 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:10:36.586+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e658 v658) 2019-12-12T00:10:37.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:10:37.506+0000 34f8d700 -1 osd.0 659 build_incremental_map_msg missing incremental map 658 2019-12-12T00:11:04.085 INFO:teuthology.orchestra.run.smithi079.stderr:testZeroByteFile (test.functional.tests.TestFileUTF8) ... ok 2019-12-12T00:11:08.070 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:38:55.003 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.078 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:38:55.011 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.096 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:38:55.002 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.100 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:38:55.033 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.102 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:38:55.008 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.104 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:38:55.038 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.386 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:38:55.292 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.399 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:38:55.333 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:11:08.436 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:38:55.369 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:12:01.579 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:01.576+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:02.607 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:02.606+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:03.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:03.617+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:04.663 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:04.660+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:05.658 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:05.657+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:06.695 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:06.694+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:07.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:07.741+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:08.772 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:08.770+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:09.807 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:09.802+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:10.775 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:10.773+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:10.948 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:10.946+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:11.228 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:11.226+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:11.254 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:11.250+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:11.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:11.725+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:11.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:11.728+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:11.950 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:11.947+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:12.283 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:12.263+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:12.284 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:12.261+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:12.385 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:12.382+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:12.707 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:12.704+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:12.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:12.707+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:12.930 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:12.926+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:13.272 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:13.269+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:13.282 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:13.280+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:13.363 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:13.360+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:13.711 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:13.708+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:13.714 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:13.713+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:13.903 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:13.901+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:13.945 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:13.941+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:14.277 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:14.275+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:14.307 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:14.304+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:14.332 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:14.329+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:14.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:14.704+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:14.708 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:14.707+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:14.924 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:14.922+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:14.955 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:14.953+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:15.249 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:15.246+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:15.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:15.308+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:15.315 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:15.314+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:15.699 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:15.696+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:15.709 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:15.701+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:15.940 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:15.938+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:15.977 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:15.975+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:16.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:16.210+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:16.352 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:16.347+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:16.353 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:16.351+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:16.696 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:16.693+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:16.699 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:16.698+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:16.921 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:16.919+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:17.005 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:17.001+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:17.238 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:17.237+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:17.364 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:17.363+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:17.379 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:17.378+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:17.751 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:17.749+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:17.752 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:17.751+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:17.960 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:17.955+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:18.036 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:18.034+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:18.232 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:18.229+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:18.383 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:18.381+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:18.418 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:18.415+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:18.768 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:18.766+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:18.771 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:18.770+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:18.981 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:18.978+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:19.074 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:19.071+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:19.261 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:19.254+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:19.392 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:19.391+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:19.468 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:19.465+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:19.759 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:19.757+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:19.760 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:19.758+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:19.974 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:19.972+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:20.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:20.100+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:20.263 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:20.260+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:20.430 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:20.428+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:20.475 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:20.472+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:20.752 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:20.749+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:20.753 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:20.752+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:20.947 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:20.943+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:21.062 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:21.060+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:21.244 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:21.241+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:21.429 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:21.427+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:21.476 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:21.474+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:21.740 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:21.738+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:21.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:21.742+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:21.975 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:21.972+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:22.079 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:22.078+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:22.280 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:22.278+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:22.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:22.456+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:22.520 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:22.517+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:22.774 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:22.765+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:22.775 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:22.768+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:22.974 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:22.968+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:23.099 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:23.097+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:23.330 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:23.327+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:23.483 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:23.480+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:23.509 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:23.502+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:23.736 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:23.733+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:23.738 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:23.736+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:23.986 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:23.984+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:24.130 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:24.127+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:24.351 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:24.348+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:24.474 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:24.467+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:24.474 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:24.467+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:24.744 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:24.742+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:24.747 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:24.747+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:24.964 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:24.959+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:25.087 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:25.085+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:25.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:25.332+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:25.485 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:25.483+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:25.494 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:25.492+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:25.722 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:25.718+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:25.724 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:25.723+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:25.971 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:25.965+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:26.107 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:26.104+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:26.386 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:26.384+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:26.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:26.454+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:26.542 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:26.538+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:26.736 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:26.734+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:26.737 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:26.737+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:26.957 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:26.953+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:27.139 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:27.137+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:27.360 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:27.358+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:27.435 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:27.432+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:27.587 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:27.584+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:27.718 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:27.716+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:27.719 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:27.718+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:27.913 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:27.907+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:28.105 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:28.102+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:28.340 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:28.338+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:28.445 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:28.443+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:28.561 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:28.559+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:28.678 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:28.676+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:28.679 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:28.678+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:28.935 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:28.934+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:29.144 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:29.142+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:29.315 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:29.313+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:29.482 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:29.479+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:29.596 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:29.592+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:29.665 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:29.650+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:29.667 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:29.652+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:29.900 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:29.899+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:30.178 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:30.175+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:30.274 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:30.272+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:30.449 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:30.446+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:30.579 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:30.576+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:30.614 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:30.612+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:30.615 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:30.615+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:30.908 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:30.906+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:31.226 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:31.223+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:31.311 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:31.309+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:31.487 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:31.485+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:31.614 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:31.611+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:31.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:31.615+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:31.625 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:31.625+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:31.919 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:31.917+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:32.186 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:32.185+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:32.342 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:32.341+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:32.483 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:32.481+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:32.643 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:32.641+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:32.643 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:32.641+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:32.645 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:32.644+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:32.926 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:32.924+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:33.149 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:33.147+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:33.343 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:33.342+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:33.460 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:33.458+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:33.661 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:33.657+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:33.661 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:33.658+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:33.662 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:33.661+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:33.969 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:33.967+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:34.125 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:34.122+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:34.370 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:34.368+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:34.433 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:34.430+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:34.668 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:34.664+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:34.668 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:34.667+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:34.711 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:34.710+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:35.023 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:35.020+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:35.097 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:35.094+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:35.409 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:35.404+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:35.453 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:35.450+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:35.657 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:35.654+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:35.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:35.657+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:35.703 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:35.701+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:36.015 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:36.011+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:36.148 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:36.146+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:36.423 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:36.420+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:36.467 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:36.464+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:36.628 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:36.625+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:36.629 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:36.627+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:36.661 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:36.657+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:37.057 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:37.053+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:37.192 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:37.191+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:37.472 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:37.470+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:37.514 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:37.511+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:37.657 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:37.655+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:37.660 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:37.660+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:37.670 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:37.668+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:38.035 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:38.031+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:38.211 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:38.209+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:38.506 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:38.503+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:38.532 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:38.530+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:38.635 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:38.631+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:38.665 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:38.662+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:38.668 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:38.664+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:39.049 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:39.045+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:39.259 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:39.256+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:39.519 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:39.515+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:39.561 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:39.557+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:39.604 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:39.600+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:39.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:39.650+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:39.656 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:39.655+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:40.072 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:40.069+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:40.283 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:40.281+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:40.542 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:40.538+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:40.573 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:40.569+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:40.574 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:40.571+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:40.650 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:40.647+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:40.651 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:40.650+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:41.114 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:41.112+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:41.334 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:41.331+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:41.534 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:41.531+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:41.535 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:41.532+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:41.586 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:41.582+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:41.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:41.614+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:41.619 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:41.617+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:42.130 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:42.128+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:42.329 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:42.327+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:42.538 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:42.535+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:42.543 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:42.541+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:42.572 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:42.568+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:42.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:42.600+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:42.605 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:42.604+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:43.183 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:43.180+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:43.345 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:43.342+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:43.522 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:43.519+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:43.545 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:43.541+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:43.580 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:43.576+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:43.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:43.657+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:43.675 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:43.674+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:44.176 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:44.173+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:44.393 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:44.390+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:44.548 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:44.544+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:44.574 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:44.570+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:44.589 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:44.586+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:44.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:44.662+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:44.669 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:44.665+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:45.205 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:45.203+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:45.372 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:45.371+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:45.503 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:45.500+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:45.566 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:45.563+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:45.603 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:45.598+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:45.705 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:45.701+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:45.706 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:45.704+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:46.242 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:46.239+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:46.329 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:12:46.325+0000 1694f700 -1 osd.3 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:24.861721+0000 front 2019-12-12T00:11:30.771014+0000 (oldest deadline 2019-12-12T00:12:10.740284+0000) 2019-12-12T00:12:46.462 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:12:46.459+0000 1694f700 -1 osd.4 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:28.708705+0000 front 2019-12-12T00:11:28.991645+0000 (oldest deadline 2019-12-12T00:12:10.870922+0000) 2019-12-12T00:12:46.562 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:12:46.559+0000 1694f700 -1 osd.2 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:26.348890+0000 front 2019-12-12T00:11:26.358991+0000 (oldest deadline 2019-12-12T00:12:11.037377+0000) 2019-12-12T00:12:46.619 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:12:46.617+0000 1694f700 -1 osd.1 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:30.043104+0000 front 2019-12-12T00:11:29.769112+0000 (oldest deadline 2019-12-12T00:12:11.968118+0000) 2019-12-12T00:12:46.755 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:46.753+0000 1694f700 -1 osd.7 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:31.034371+0000 front 2019-12-12T00:11:27.424765+0000 (oldest deadline 2019-12-12T00:12:10.849185+0000) 2019-12-12T00:12:46.759 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:46.758+0000 1694f700 -1 osd.7 665 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:47.179 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:12:47.176+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 45sec e665 v665) 2019-12-12T00:12:47.217 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:12:47.213+0000 1694f700 -1 osd.6 665 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:11:29.679291+0000 front 2019-12-12T00:11:30.043919+0000 (oldest deadline 2019-12-12T00:12:13.048677+0000) 2019-12-12T00:12:48.551 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:48.528+0000 1694f700 -1 osd.7 666 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:12:49.658 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:12:49.651+0000 1694f700 -1 osd.7 667 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122605 2.27s0 2:e675b977:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_a:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e665) 2019-12-12T00:13:20.961 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:20.910+0000 31786700 -1 osd.2 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.086 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:21.039+0000 31f87700 -1 osd.2 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.125 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:21.040+0000 31f87700 -1 osd.2 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.162 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:21.068+0000 35f8f700 -1 osd.2 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.224 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:21.089+0000 35f8f700 -1 osd.2 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.277 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:21.169+0000 31786700 -1 osd.2 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.473 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:21.355+0000 36790700 -1 osd.6 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.595 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:21.402+0000 37f93700 -1 osd.6 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.731 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:21.450+0000 37f93700 -1 osd.6 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.819 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:21.585+0000 3478c700 -1 osd.6 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:21.881 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:21.613+0000 3478c700 -1 osd.6 669 build_incremental_map_msg missing incremental map 669 2019-12-12T00:13:22.610 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:41:09.543 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.626 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:41:09.559 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.630 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:41:09.564 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.640 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:41:09.573 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.720 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:41:09.626 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.726 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:41:09.632 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.809 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:41:09.743 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.899 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:41:09.805 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:22.934 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:41:09.867 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:25.189 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:24.872+0000 36790700 -1 osd.0 671 build_incremental_map_msg missing incremental map 671 2019-12-12T00:13:25.205 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:24.916+0000 31f87700 -1 osd.0 671 build_incremental_map_msg missing incremental map 671 2019-12-12T00:13:27.883 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:27.873+0000 32f89700 -1 osd.0 671 build_incremental_map_msg missing incremental map 671 2019-12-12T00:13:37.760 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:41:24.693 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.777 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:41:24.710 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.795 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:41:24.729 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.798 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:41:24.731 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.838 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:41:24.744 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.838 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:41:24.772 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.839 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:41:24.773 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.841 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:41:24.747 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:37.888 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:41:24.794 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:13:45.763 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:45.761+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:46.435 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:46.433+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:46.602 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:46.599+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:46.781 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:46.778+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:47.425 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:47.422+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:47.489 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:47.487+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:47.607 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:47.606+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:47.830 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:47.827+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:48.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:48.461+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:48.489 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:48.487+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:48.608 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:48.606+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:48.848 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:48.846+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:49.479 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:49.476+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:49.479 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:49.477+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:49.541 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:49.539+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:49.658 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:49.656+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:49.896 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:49.894+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:50.455 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:50.452+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:50.506 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:50.502+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:50.577 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:50.573+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:50.709 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:50.706+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:50.876 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:50.873+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:51.420 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:51.414+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:51.553 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:51.551+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:51.564 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:51.563+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:51.691 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:51.690+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:51.838 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:51.835+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:52.441 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:52.439+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:52.604 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:52.601+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:52.606 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:52.605+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:52.723 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:52.721+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:52.763 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:52.761+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:122811 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 821560752 gen 122] snapc 0=[] ondisk+write+known_if_redirected e669) 2019-12-12T00:13:52.859 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:52.857+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:53.416 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:53.413+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:53.565 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:53.563+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:53.634 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:53.632+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:53.744 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:53.739+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:53.757 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:53.756+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:13:53.875 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:53.874+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:54.459 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:54.457+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:54.532 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:54.531+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:54.599 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:54.596+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:54.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:54.754+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:13:54.757 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:54.756+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:54.859 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:54.858+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:55.438 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:55.427+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:55.530 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:55.520+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:55.616 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:55.615+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:55.720 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:55.719+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:13:55.759 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:55.757+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:55.869 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:55.867+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:56.476 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:56.474+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:56.481 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:56.479+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:56.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:56.580+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:56.711 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:56.709+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:13:56.776 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:56.774+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:56.825 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:56.823+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:57.474 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:57.472+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:57.479 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:57.473+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:57.549 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:57.547+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:57.703 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:57.702+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:13:57.772 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:57.768+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:57.828 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:57.826+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:58.489 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:58.488+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:58.493 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:58.490+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:58.558 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:58.556+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:58.744 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:58.743+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:13:58.773 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:58.771+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:58.874 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:58.873+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:13:59.466 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:13:59.464+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:13:59.497 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:13:59.495+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:13:59.515 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:13:59.513+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:13:59.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:59.705+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:13:59.709 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:13:59.707+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:13:59.798 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:13:59.794+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:13:59.850 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:13:59.848+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:00.465 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:00.459+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:00.500 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:14:00.498+0000 1694f700 -1 osd.7 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:03.488715+0000 front 2019-12-12T00:13:03.497542+0000 (oldest deadline 2019-12-12T00:13:46.099413+0000) 2019-12-12T00:14:00.537 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:00.536+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:00.738 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:00.729+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:00.739 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:00.731+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:00.813 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:00.811+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:00.889 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:00.888+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:01.450 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:01.447+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:01.570 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:01.568+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:01.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:01.734+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:01.736 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:01.734+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:01.784 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:01.783+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:01.874 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:01.871+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:02.461 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:02.458+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:02.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:02.580+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:02.757 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:02.754+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:02.768 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:02.767+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:02.770 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:02.770+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:02.880 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:02.879+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:03.491 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:03.488+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:03.604 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:03.601+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:03.742 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:03.740+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:03.756 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:03.752+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:03.757 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:03.755+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:03.854 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:03.852+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:04.500 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:04.498+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:04.580 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:04.577+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:04.750 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:04.748+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:04.763 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:04.760+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:04.764 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:04.762+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:04.814 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:04.810+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:05.518 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:05.516+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:05.627 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:05.625+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:05.727 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:05.725+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:05.771 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:05.767+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:05.807 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:05.805+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:05.808 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:05.806+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:06.517 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:06.515+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:06.602 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:06.599+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:06.716 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:06.713+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:06.792 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:06.789+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:06.860 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:06.856+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:06.860 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:06.858+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:07.482 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:07.480+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:07.608 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:07.605+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:07.679 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:07.677+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:07.817 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:07.814+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:07.819 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:07.816+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:07.822 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:07.820+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:08.496 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:08.494+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:08.625 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:08.622+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:08.678 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:08.675+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:08.785 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:08.781+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:08.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:08.808+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:08.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:08.809+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:09.515 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:09.513+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:09.641 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:09.638+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:09.720 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:09.718+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:09.745 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:09.741+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:09.844 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:09.841+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:09.844 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:09.842+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:10.527 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:10.524+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:10.678 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:10.675+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:10.740 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:10.738+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:10.750 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:10.748+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:10.810 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:10.807+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:10.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:10.809+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:11.508 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:11.504+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:11.681 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:11.679+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:11.745 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:11.743+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:11.757 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:11.754+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:11.847 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:11.844+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:11.850 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:11.848+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:12.534 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:12.532+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:12.677 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:12.673+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:12.745 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:12.742+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:12.761 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:12.758+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:12.886 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:12.883+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:12.886 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:12.884+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:13.573 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:13.571+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:13.695 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:13.693+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:13.723 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:13.720+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:13.760 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:13.758+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:13.931 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:13.928+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:13.932 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:13.930+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:14.619 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:14.617+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:14.672 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:14.671+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:14.748 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:14.747+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:14.800 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:14.798+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:14.902 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:14.899+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:14.903 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:14.902+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:15.603 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:15.601+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:15.711 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:15.708+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:15.715 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:15.712+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:15.826 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:15.824+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:15.922 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:15.919+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:15.922 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:15.921+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:16.649 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:16.643+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:16.677 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:16.675+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:16.726 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:16.722+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:16.851 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:16.834+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:16.929 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:16.927+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:16.930 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:16.928+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:17.647 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:17.644+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:17.676 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:17.674+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:17.713 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:17.710+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:17.831 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:14:17.828+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e671 v671) 2019-12-12T00:14:17.872 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:17.870+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:17.914 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:17.912+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:17.915 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:17.914+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:18.639 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:18.636+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:18.684 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:18.681+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:18.766 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:18.762+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:18.871 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:18.869+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:18.873 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:18.872+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:18.876 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:18.874+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:19.669 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:19.667+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:19.683 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:19.680+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:19.735 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:19.732+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:19.828 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:19.825+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:19.829 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:19.827+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:19.911 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:19.909+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:20.657 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:20.657+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:20.716 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:20.714+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:20.735 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:20.733+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:20.796 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:20.792+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:20.796 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:20.794+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:20.891 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:20.890+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:21.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:21.633+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:21.695 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:21.693+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:21.784 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:21.782+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:21.828 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:21.826+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:21.829 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:21.827+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:21.867 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:21.866+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:22.655 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:22.653+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:22.663 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:22.663+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:22.750 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:22.747+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:22.805 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:22.803+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:22.808 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:22.806+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:22.828 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:22.828+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:22.846 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:14:22.843+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e671 v671) 2019-12-12T00:14:23.639 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:23.638+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:23.696 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:23.693+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:23.788 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:23.785+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:23.792 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:23.789+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:23.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:23.791+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:23.827 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:23.823+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:24.599 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:24.596+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:24.708 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:24.707+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:24.814 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:24.813+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:24.829 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:24.826+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:24.844 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:24.843+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:24.848 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:24.846+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:25.591 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:25.589+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:25.748 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:25.746+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:25.802 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:25.800+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:25.867 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:25.866+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:25.883 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:25.880+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:25.884 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:25.882+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:26.615 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:26.597+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:26.801 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:26.796+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:26.831 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:26.830+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:26.881 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:26.880+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:26.891 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:26.888+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:26.891 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:26.890+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:27.586 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:27.584+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:27.824 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:27.823+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:27.825 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:27.824+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:27.863 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:14:27.860+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e671 v671) 2019-12-12T00:14:27.894 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:27.892+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:27.895 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:27.894+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:27.904 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:27.902+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:28.617 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:28.614+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:28.787 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:28.784+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:28.814 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:28.812+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:28.848 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:28.846+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:28.850 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:28.848+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:28.954 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:28.951+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:29.629 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:29.627+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:29.754 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:29.752+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:29.843 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:29.841+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:29.843 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:29.842+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:29.845 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:29.843+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:29.918 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:29.916+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:30.674 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:30.672+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:30.752 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:30.751+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:30.829 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:30.826+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:30.876 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:30.874+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:30.877 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:30.875+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:30.945 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:30.942+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:31.727 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:31.725+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:31.765 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:31.764+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:31.865 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:31.862+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:31.888 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:31.886+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:31.891 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:31.890+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:31.935 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:31.932+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:32.705 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:32.703+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:32.765 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:32.762+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:32.862 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:32.861+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:32.880 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:14:32.878+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e671 v671) 2019-12-12T00:14:32.892 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:32.891+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:32.895 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:32.892+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:32.971 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:32.969+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:33.697 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:33.695+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:33.779 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:33.776+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:33.820 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:33.817+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:33.942 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:33.940+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:33.943 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:33.942+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:33.974 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:33.972+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:34.701 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:34.699+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:34.763 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:34.761+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:34.793 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:34.792+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:34.969 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:34.967+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:34.970 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:34.969+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:35.021 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:35.019+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:35.745 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:35.743+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:35.767 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:35.764+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:35.778 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:35.776+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:35.982 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:35.980+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:35.984 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:35.982+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:36.064 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:36.062+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:36.755 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:36.753+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:36.782 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:36.780+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:36.812 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:36.810+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:36.999 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:36.996+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:36.999 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:36.998+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:37.071 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:37.069+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:37.735 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:14:37.732+0000 1694f700 -1 osd.6 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:05.150270+0000 front 2019-12-12T00:13:05.108570+0000 (oldest deadline 2019-12-12T00:13:47.119883+0000) 2019-12-12T00:14:37.768 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:14:37.766+0000 1694f700 -1 osd.2 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:04.503005+0000 front 2019-12-12T00:13:04.514445+0000 (oldest deadline 2019-12-12T00:13:49.171726+0000) 2019-12-12T00:14:37.806 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:14:37.804+0000 1694f700 -1 osd.1 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:01.597047+0000 front 2019-12-12T00:13:01.595309+0000 (oldest deadline 2019-12-12T00:13:45.085313+0000) 2019-12-12T00:14:37.933 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:14:37.930+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e671 v671) 2019-12-12T00:14:37.998 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:37.996+0000 1694f700 -1 osd.0 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:13:19.256621+0000 (oldest deadline 2019-12-12T00:13:59.256621+0000) 2019-12-12T00:14:37.999 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:37.998+0000 1694f700 -1 osd.0 671 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:38.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:14:38.065+0000 1694f700 -1 osd.4 671 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:13:02.712526+0000 front 2019-12-12T00:13:02.616796+0000 (oldest deadline 2019-12-12T00:13:46.582010+0000) 2019-12-12T00:14:39.042 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:38.991+0000 1694f700 -1 osd.0 672 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e669) 2019-12-12T00:14:39.307 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:14:39.290+0000 38f95700 -1 osd.0 672 build_incremental_map_msg missing incremental map 671 2019-12-12T00:15:02.675 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:42:49.607 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.693 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:42:49.623 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.704 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:42:49.636 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.709 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:42:49.640 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.715 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:42:49.620 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.730 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:42:49.634 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.816 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:42:49.748 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.867 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:42:49.799 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:02.922 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:42:49.828 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:15:58.122 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:15:58.121+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:15:59.094 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:15:59.085+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:00.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:00.085+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:01.127 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:01.124+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:02.093 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:02.092+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:03.068 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:03.067+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:04.118 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:04.118+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:05.081 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:05.080+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:06.108 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:06.107+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:07.126 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:07.125+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:08.158 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:08.156+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:09.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:09.171+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:10.120 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:10.117+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:10.210 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:10.209+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:11.142 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:11.138+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:11.183 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:11.179+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:11.184 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:11.182+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:11.447 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:11.444+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:11.566 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:11.565+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:12.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:12.105+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:12.186 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:12.180+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:12.187 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:12.185+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:12.475 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:12.472+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:12.583 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:12.581+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:13.112 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:13.111+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:13.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:13.176+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:13.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:13.181+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:13.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:13.511+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:13.539 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:13.536+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:13.636 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:13.634+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:14.147 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:14.145+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:14.184 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:14.183+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:14.186 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:14.187+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:14.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:14.211+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:14.504 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:14.502+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:14.527 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:14.525+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:14.642 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:14.640+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:15.164 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:15.161+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:15.240 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:15.237+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:15.241 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:15.239+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:15.258 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:15.256+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:15.495 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:15.492+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:15.562 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:15.560+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:15.675 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:15.673+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:16.199 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:16.196+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:16.259 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:16.257+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:16.259 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:16.259+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:16.315 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:16.314+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:16.531 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:16.529+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:16.533 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:16.531+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:16.716 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:16.713+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:17.165 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:17.164+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:17.223 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:17.219+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:17.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:17.222+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:17.367 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:17.366+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:17.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:17.491+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:17.550 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:17.546+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:17.681 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:17.679+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:18.134 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:18.130+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:18.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:18.270+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:18.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:18.275+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:18.357 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:18.355+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:18.482 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:18.479+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:18.525 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:18.516+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:18.645 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:18.644+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:19.147 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:19.144+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:19.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:19.248+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:19.251 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:19.251+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:19.378 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:19.376+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:19.498 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:19.494+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:19.528 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:19.524+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:19.689 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:19.684+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:20.182 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:20.178+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:20.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:20.275+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:20.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:20.278+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:20.412 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:20.410+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:20.476 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:20.472+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:20.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:20.576+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:20.730 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:20.728+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:21.224 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:21.211+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:21.315 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:21.315+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:21.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:21.319+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:21.432 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:21.426+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:21.462 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:21.460+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:21.591 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:21.587+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:21.686 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:21.683+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:22.192 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:22.188+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:22.343 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:22.340+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:22.344 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:22.342+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:22.438 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:22.435+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:22.475 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:22.473+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:22.545 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:22.543+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:22.647 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:22.644+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:23.162 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:23.159+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:23.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:23.295+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:23.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:23.297+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:23.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:23.471+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:23.503 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:23.500+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:23.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:23.581+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:23.657 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:23.655+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:24.136 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:24.133+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:24.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:24.278+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:24.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:24.280+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:24.485 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:24.482+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:24.548 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:24.546+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:24.618 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:24.616+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:24.631 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:24.630+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:25.139 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:25.137+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:25.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:25.296+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:25.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:25.298+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:25.450 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:25.446+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:25.575 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:25.572+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:25.627 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:25.624+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:25.646 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:25.643+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:26.109 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:26.107+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:26.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:26.300+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:26.304 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:26.303+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:26.473 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:26.471+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:26.577 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:26.575+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:26.616 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:26.614+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:26.660 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:26.658+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:27.062 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:27.060+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:27.352 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:27.349+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:27.353 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:27.351+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:27.503 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:27.500+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:27.579 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:27.577+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:27.656 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:27.653+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:27.679 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:27.676+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:28.032 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:28.031+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:28.339 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:28.337+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:28.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:28.341+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:28.525 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:28.523+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:28.579 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:28.575+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:28.692 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:28.690+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:28.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:28.719+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:29.074 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:29.072+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:29.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:29.310+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:29.318 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:29.317+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:29.513 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:29.507+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:29.614 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:29.609+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:29.681 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:29.678+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:29.752 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:29.751+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:30.117 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:30.115+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:30.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:30.357+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:30.363 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:30.362+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:30.473 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:30.471+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:30.633 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:30.629+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:30.734 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:30.728+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:30.740 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:30.738+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:31.146 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:31.144+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:31.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:31.405+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:31.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:31.408+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:31.471 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:31.469+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:31.649 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:31.646+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:31.720 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:31.718+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:31.778 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:31.775+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:32.163 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:32.161+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:32.447 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:32.444+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:32.453 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:32.447+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:32.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:32.503+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:32.638 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:32.635+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:32.723 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:32.720+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:32.786 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:32.783+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:33.209 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:33.204+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:33.467 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:33.465+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:33.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:33.470+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:33.546 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:33.542+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:33.622 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:33.620+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:33.737 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:33.735+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:33.782 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:33.779+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:34.223 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:34.219+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:34.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:34.468+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:34.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:34.471+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:34.531 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:34.528+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:34.640 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:34.637+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:34.729 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:34.728+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:34.758 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:34.756+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:35.263 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:35.260+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:35.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:35.425+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:35.429 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:35.428+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:35.528 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:35.526+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:35.687 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:35.685+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:35.732 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:35.729+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:35.788 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:35.785+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:36.244 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:36.243+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:36.439 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:36.433+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:36.441 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:36.439+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:36.557 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:36.552+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:36.731 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:36.728+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:36.744 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:36.742+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:36.764 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:36.761+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:37.204 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:37.202+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:37.486 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:37.483+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:37.487 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:37.486+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:37.518 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:37.516+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:37.705 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:37.702+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:37.761 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:37.758+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:37.810 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:37.808+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:38.214 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:38.211+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:38.472 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:38.470+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:38.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:38.474+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:38.481 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:38.479+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:38.746 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:38.744+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:38.750 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:38.748+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:38.771 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:38.768+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:39.203 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:39.201+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:39.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:39.460+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:39.464 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:39.463+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:39.507 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:39.505+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:39.755 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:39.752+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:39.764 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:39.761+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:39.765 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:39.763+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:40.251 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:40.249+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:40.483 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:40.480+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:40.484 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:40.483+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:40.554 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:40.551+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:40.743 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:40.740+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:40.758 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:40.755+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:40.767 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:40.762+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:41.249 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:41.247+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:41.470 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:41.467+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:41.471 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:41.470+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:41.526 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:41.522+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:41.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:41.735+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:41.794 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:41.792+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:41.806 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:41.803+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:42.229 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:42.226+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:42.486 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:42.484+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:42.501 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:42.498+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:42.502 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:42.501+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:42.770 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:42.767+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:42.790 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:42.788+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:42.803 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:42.801+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:43.259 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:43.257+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:43.459 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:43.456+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:43.489 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:16:43.485+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 44sec e677 v677) 2019-12-12T00:16:43.568 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:43.556+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:43.570 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:43.559+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:43.731 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:43.728+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:43.748 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:43.745+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:43.856 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:43.853+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:44.278 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:44.277+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:44.456 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:44.455+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:44.583 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:44.579+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:44.583 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:44.582+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:44.754 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:44.750+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:44.800 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:44.799+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:44.888 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:44.885+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:45.251 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:45.250+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:45.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:45.498+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:45.581 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:45.579+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:45.584 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:45.584+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:45.726 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:45.724+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:45.796 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:45.794+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:45.878 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:45.875+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:46.296 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:46.294+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:46.482 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:46.481+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:46.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:46.586+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:46.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:46.590+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:46.740 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:46.738+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:46.751 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:46.749+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:46.921 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:46.914+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:47.340 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:47.338+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:47.445 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:47.443+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:47.589 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:47.586+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:47.590 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:47.589+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:47.753 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:16:47.750+0000 1694f700 -1 osd.6 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.360313+0000 front 2019-12-12T00:15:28.346765+0000 (oldest deadline 2019-12-12T00:16:13.619828+0000) 2019-12-12T00:16:47.760 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:16:47.758+0000 1694f700 -1 osd.2 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.120494+0000 front 2019-12-12T00:15:29.780058+0000 (oldest deadline 2019-12-12T00:16:11.403007+0000) 2019-12-12T00:16:47.884 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:16:47.881+0000 1694f700 -1 osd.4 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:28.292382+0000 front 2019-12-12T00:15:28.290624+0000 (oldest deadline 2019-12-12T00:16:14.159852+0000) 2019-12-12T00:16:48.397 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:16:48.393+0000 1694f700 -1 osd.1 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:29.385233+0000 front 2019-12-12T00:15:26.104695+0000 (oldest deadline 2019-12-12T00:16:09.345203+0000) 2019-12-12T00:16:48.465 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:16:48.463+0000 1694f700 -1 osd.3 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:25.623932+0000 front 2019-12-12T00:15:25.624478+0000 (oldest deadline 2019-12-12T00:16:10.917346+0000) 2019-12-12T00:16:48.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:48.556+0000 1694f700 -1 osd.7 677 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:15:26.139996+0000 front 2019-12-12T00:15:26.354909+0000 (oldest deadline 2019-12-12T00:16:10.212319+0000) 2019-12-12T00:16:48.560 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:48.559+0000 1694f700 -1 osd.7 677 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:48.563 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:16:48.561+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 44sec e677 v677) 2019-12-12T00:16:49.731 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:49.708+0000 1694f700 -1 osd.7 678 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:50.905 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:50.898+0000 1694f700 -1 osd.7 679 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:51.943 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:51.923+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:52.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:52.935+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:53.902 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:53.901+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:54.909 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:54.908+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:55.924 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:55.923+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:56.973 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:56.972+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:57.982 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:57.982+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:16:59.016 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:16:59.012+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:00.047 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:00.046+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:01.048 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:01.047+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:02.067 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:02.065+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:03.093 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:03.092+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:04.101 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:04.100+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:05.092 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:05.091+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:06.052 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:06.051+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:07.052 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:07.051+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:08.039 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:08.035+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:09.000 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:08.998+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:10.025 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:10.024+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:11.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:11.072+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:12.095 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:12.095+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:13.114 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:13.115+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:14.138 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:14.139+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:15.132 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:15.132+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:16.167 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:16.167+0000 1694f700 -1 osd.7 680 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:17.217 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:17.216+0000 1694f700 -1 osd.7 681 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:18.593 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:18.592+0000 1694f700 -1 osd.7 682 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:19.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:19.554+0000 1694f700 -1 osd.7 683 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:20.793 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:20.755+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:21.777 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:21.776+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:22.795 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:22.785+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:23.530 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:23.520+0000 3478c700 -1 osd.0 684 build_incremental_map_msg missing incremental map 684 2019-12-12T00:17:23.830 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:23.830+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:23.984 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:23.941+0000 32f89700 -1 osd.0 684 build_incremental_map_msg missing incremental map 684 2019-12-12T00:17:24.883 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:24.882+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:25.895 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:25.893+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:26.879 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:26.878+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:27.862 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:27.861+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:28.910 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:28.910+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:29.968 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:29.966+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:30.989 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:30.988+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:32.000 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:31.999+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:32.975 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:32.973+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:33.949 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:33.948+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:34.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:34.977+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:35.953 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:35.952+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:36.991 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:36.990+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:38.016 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:38.015+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:38.980 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:38.979+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:40.016 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:40.012+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:40.983 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:40.981+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:41.947 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:41.947+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:42.977 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:42.976+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:43.971 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:43.969+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:44.946 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:44.945+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:45.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:45.942+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:46.982 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:46.980+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:47.996 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:47.994+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:47.998 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:47.998+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:48.393 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:48.391+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:48.400 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:48.398+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:48.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:48.989+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:48.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:48.991+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:49.358 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:49.355+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:49.448 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:49.444+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:49.680 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:49.676+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:49.715 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:49.712+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:49.753 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:49.751+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:49.989 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:49.988+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:49.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:49.991+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:50.346 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:50.344+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:50.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:50.462+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:50.658 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:50.656+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:50.738 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:50.735+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:50.772 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:50.770+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:50.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:50.976+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:50.981 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:50.980+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:51.363 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:51.361+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:51.472 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:51.469+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:51.651 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:51.648+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:51.726 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:51.723+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:51.733 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:51.731+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:51.937 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:51.935+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:51.940 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:51.939+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:52.338 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:52.329+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:52.485 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:52.484+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:52.659 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:52.657+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:52.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:52.696+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:52.758 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:52.756+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:52.977 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:52.975+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:52.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:52.977+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:53.301 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:53.299+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:53.508 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:53.505+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:53.657 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:53.655+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:53.703 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:53.701+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:53.764 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:53.762+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:53.989 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:53.987+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:53.993 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:53.989+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:54.339 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:54.336+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:54.495 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:54.493+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:54.672 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:54.671+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:54.708 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:54.706+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:54.771 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:54.770+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:55.042 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:55.039+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:55.042 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:55.041+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:55.347 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:55.345+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:55.551 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:55.548+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:55.631 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:55.628+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:55.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:55.696+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:55.734 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:55.733+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:56.062 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:56.060+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:56.063 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:56.062+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:56.330 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:56.328+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:56.538 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:56.536+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:56.677 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:56.675+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:56.700 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:56.698+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:17:56.700 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:56.699+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:56.701 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:56.697+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:57.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:57.069+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:57.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:57.071+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:57.332 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:57.330+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:57.595 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:57.592+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:57.681 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:57.678+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:17:57.682 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:57.680+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:57.698 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:57.694+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:57.699 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:57.698+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:58.040 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:58.035+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:58.040 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:58.039+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:58.344 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:58.342+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:58.611 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:58.609+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:58.697 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:58.695+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:58.712 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:58.709+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:17:58.713 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:58.710+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:17:58.715 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:58.713+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:59.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:59.069+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:17:59.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:17:59.071+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:17:59.352 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:17:59.349+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:17:59.639 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:17:59.632+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:17:59.667 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:17:59.666+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:17:59.690 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:17:59.689+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:17:59.735 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:59.734+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:17:59.736 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:17:59.736+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:00.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:00.071+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:00.075 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:00.074+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:00.331 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:00.329+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:00.651 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:00.648+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:00.663 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:00.661+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:00.692 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:00.690+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:00.747 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:00.746+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:00.748 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:00.747+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:01.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:01.068+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:01.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:01.070+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:01.358 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:01.353+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:01.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:01.634+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:01.645 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:01.643+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:01.699 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:01.697+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:01.777 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:01.775+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:01.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:01.776+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:02.051 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:02.048+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:02.051 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:02.049+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:02.382 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:02.379+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:02.652 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:02.651+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:02.666 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:02.664+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:02.698 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:02.696+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:02.804 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:02.801+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:02.804 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:02.802+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:03.080 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:03.077+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:03.080 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:03.079+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:03.413 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:03.411+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:03.635 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:03.633+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:03.698 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:03.696+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:03.704 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:03.702+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:03.817 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:03.814+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:03.817 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:03.815+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:04.116 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:04.114+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:04.119 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:04.118+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:04.431 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:04.429+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:04.623 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:04.620+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:04.728 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:04.727+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:04.738 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:04.736+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:04.849 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:04.847+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:04.849 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:04.848+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:05.161 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:05.159+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:05.163 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:05.163+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:05.407 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:05.406+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:05.648 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:05.646+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:05.712 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:05.710+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:05.758 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:05.755+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:05.810 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:05.807+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:05.811 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:05.809+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:06.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:06.187+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:06.191 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:06.189+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:06.461 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:06.459+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:06.652 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:06.650+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:06.732 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:06.730+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:06.806 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:06.804+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:06.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:06.810+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:06.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:06.812+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:07.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:07.220+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:07.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:07.224+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:07.432 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:07.430+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:07.660 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:07.659+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:07.781 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:07.778+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:07.825 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:07.823+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:07.827 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:07.826+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:07.851 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:07.849+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:08.236 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:08.232+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:08.242 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:08.241+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:08.443 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:08.443+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:08.623 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:08.622+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:08.737 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:08.735+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:08.791 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:08.788+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:08.792 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:08.790+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:08.903 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:08.901+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:09.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:09.292+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:09.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:09.294+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:09.453 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:09.453+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:09.626 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:09.623+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:09.785 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:09.783+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:09.789 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:09.788+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:09.790 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:09.790+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:09.891 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:09.890+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:10.320 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:10.319+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:10.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:10.321+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:10.493 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:10.492+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:10.590 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:10.588+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:10.778 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:10.776+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:10.831 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:10.829+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:10.834 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:10.832+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:10.936 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:10.935+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:11.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:11.292+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:11.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:11.294+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:11.460 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:11.459+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:11.582 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:11.581+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:11.761 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:11.760+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:11.823 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:11.820+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:11.823 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:11.822+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:11.941 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:11.939+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:12.255 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:12.253+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:12.255 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:12.255+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:12.503 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:12.502+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:12.584 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:12.583+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:12.726 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:12.725+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:12.832 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:12.830+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:12.833 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:12.832+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:12.949 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:12.947+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:13.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:13.220+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:13.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:13.224+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:13.468 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:13.467+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:13.585 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:13.584+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:13.703 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:13.701+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:13.863 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:13.858+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:13.863 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:13.862+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:13.932 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:13.929+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:14.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:14.257+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:14.260 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:14.261+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:14.461 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:14.461+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:14.596 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:14.595+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:14.735 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:14.734+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:14.869 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:14.866+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:14.869 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:14.868+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:14.959 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:14.957+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:15.273 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:15.271+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:15.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:15.273+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:15.492 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:15.490+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:15.558 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:15.556+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:15.709 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:15.703+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:15.837 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:15.835+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:15.838 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:15.837+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:15.920 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:15.918+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:16.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:16.253+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:16.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:16.255+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:16.538 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:16.536+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:16.605 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:16.599+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:16.664 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:16.662+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:16.800 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:16.798+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:16.802 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:16.801+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:16.903 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:16.901+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:17.236 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:17.227+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:17.239 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:17.238+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:17.557 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:17.555+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:17.595 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:17.593+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:17.706 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:17.704+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:17.810 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:17.807+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:17.810 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:17.809+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:17.871 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:17.868+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:18.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:18.285+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:18.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:18.287+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:18.535 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:18.532+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:18.646 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:18.643+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:18.730 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:18.727+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:18.786 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:18.784+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:18.792 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:18.788+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:18.899 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:18.897+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:19.300 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:19.297+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:19.301 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:19.299+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:19.534 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:19.532+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:19.692 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:19.690+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:19.729 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:19.728+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:19.812 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:19.809+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:19.813 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:19.811+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:19.913 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:19.910+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:20.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:20.308+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:20.314 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:20.311+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:20.501 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:20.499+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:20.700 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:20.699+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:20.751 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:20.748+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:20.766 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:20.764+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:20.767 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:20.765+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:20.906 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:20.905+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:21.331 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:21.328+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:21.332 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:21.330+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:21.477 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:21.476+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:21.673 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:21.669+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:21.745 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:21.743+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:21.812 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:21.806+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:21.814 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:21.811+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:21.944 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:21.942+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:22.373 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:22.371+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:22.376 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:22.375+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:22.502 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:22.500+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:22.667 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:22.666+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:22.781 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:22.778+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:22.805 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:22.802+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:22.805 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:22.804+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:22.984 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:22.980+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:23.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:23.379+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:23.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:23.384+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:23.479 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:23.477+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:23.654 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:23.651+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:23.745 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:23.743+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:23.793 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:23.792+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:23.794 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:23.792+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:23.883 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:18:23.876+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 44sec e684 v684) 2019-12-12T00:18:23.966 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:23.965+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:24.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:24.340+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:24.343 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:24.342+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:24.520 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:24.519+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:24.625 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:24.623+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:24.729 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:24.727+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:24.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:24.777+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:24.779 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:24.778+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:24.948 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:24.945+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:25.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:25.317+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:25.321 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:25.320+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:25.508 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:25.505+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:25.662 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:25.661+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:25.745 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:25.744+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:25.758 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:25.756+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:25.759 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:25.757+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:25.917 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:25.915+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:26.293 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:26.291+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:26.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:26.295+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:26.482 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:26.479+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:26.641 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:26.639+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:26.701 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:26.699+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:26.774 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:26.772+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:26.774 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:26.773+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:26.968 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:26.967+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:27.308 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:27.306+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:27.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:27.310+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:27.510 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:27.507+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:27.684 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:27.682+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:27.715 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:27.713+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:27.781 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:27.778+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:27.782 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:27.780+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:27.940 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:27.937+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:28.338 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:28.336+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:28.347 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:28.345+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:28.545 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:28.543+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:28.672 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:28.670+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:28.770 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:28.768+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:28.801 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:28.799+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:28.806 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:28.805+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:28.892 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:18:28.886+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 44sec e684 v684) 2019-12-12T00:18:28.907 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:28.905+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:29.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:29.359+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:29.363 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:29.363+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:29.532 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:29.529+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:29.657 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:29.655+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:29.769 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:29.765+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:29.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:29.775+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:29.779 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:29.777+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:29.910 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:29.906+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:30.330 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:30.327+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:30.330 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:30.329+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:30.520 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:30.517+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:30.614 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:30.612+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:30.819 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:30.817+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:30.822 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:30.820+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:30.822 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:30.820+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:30.935 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:30.931+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:31.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:31.333+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:31.336 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:31.335+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:31.568 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:31.566+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:31.637 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:31.634+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:31.842 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:31.838+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:31.866 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:31.862+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:31.867 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:31.863+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:31.957 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:31.954+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:32.357 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:32.354+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:32.358 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:32.356+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:32.555 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:32.553+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:32.642 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:32.640+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:32.854 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:32.850+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:32.855 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:32.852+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:32.858 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:32.853+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:32.957 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:32.954+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:33.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:33.406+0000 1694f700 -1 osd.7 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:03.379212+0000 front 2019-12-12T00:17:03.381264+0000 (oldest deadline 2019-12-12T00:17:47.415885+0000) 2019-12-12T00:18:33.409 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:33.408+0000 1694f700 -1 osd.7 684 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:33.565 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:18:33.562+0000 1694f700 -1 osd.4 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:04.573994+0000 front 2019-12-12T00:17:04.586057+0000 (oldest deadline 2019-12-12T00:17:47.434778+0000) 2019-12-12T00:18:33.642 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:18:33.640+0000 1694f700 -1 osd.6 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:02.401713+0000 front 2019-12-12T00:17:02.143240+0000 (oldest deadline 2019-12-12T00:17:48.026367+0000) 2019-12-12T00:18:33.892 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:33.886+0000 1694f700 -1 osd.0 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:17:16.004694+0000 (oldest deadline 2019-12-12T00:17:56.004694+0000) 2019-12-12T00:18:33.892 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:18:33.888+0000 1694f700 -1 osd.0 684 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e682) 2019-12-12T00:18:33.903 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:18:33.899+0000 1694f700 -1 osd.2 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.388975+0000 front 2019-12-12T00:17:05.625256+0000 (oldest deadline 2019-12-12T00:17:48.737392+0000) 2019-12-12T00:18:33.959 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:18:33.957+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 44sec e684 v684) 2019-12-12T00:18:34.019 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:18:34.010+0000 1694f700 -1 osd.1 684 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:17:05.160920+0000 front 2019-12-12T00:17:05.237432+0000 (oldest deadline 2019-12-12T00:17:49.174701+0000) 2019-12-12T00:18:34.952 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:18:34.907+0000 1694f700 -1 osd.7 685 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4518.0:123132 2.2fs0 2:f5acb010:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e8%94%89%e9%90%8b%e5%98%85%e1%a0%83%ef%84%90%e9%90%8a%e0%a4%83%e1%a0%84%e9%90%8b%e3%9c%83%e1%a0%84%e5%98%87%ef%84%90%e0%a4%82%ef%84%92:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e677) 2019-12-12T00:18:53.335 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:46:40.268 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.345 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:46:40.279 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.356 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:46:40.289 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.359 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:46:40.293 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.372 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:46:40.278 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.384 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:46:40.290 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.616 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:46:40.549 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.683 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:46:40.588 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:18:53.683 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:46:40.616 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:19:07.544 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:19:07.344+0000 35f8f700 -1 osd.3 690 build_incremental_map_msg missing incremental map 690 2019-12-12T00:19:07.628 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:19:07.359+0000 31f87700 -1 osd.3 690 build_incremental_map_msg missing incremental map 690 2019-12-12T00:19:07.673 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:19:07.478+0000 32f89700 -1 osd.3 690 build_incremental_map_msg missing incremental map 690 2019-12-12T00:19:08.008 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:19:07.951+0000 36f91700 -1 osd.3 690 build_incremental_map_msg missing incremental map 690 2019-12-12T00:19:08.078 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:19:08.044+0000 38f95700 -1 osd.3 690 build_incremental_map_msg missing incremental map 690 2019-12-12T00:20:05.441 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:05.438+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:06.130 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:06.126+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:06.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:06.218+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:06.393 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:06.390+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:06.485 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:06.482+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:06.567 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:06.565+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:07.176 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:07.173+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:07.268 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:07.266+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:07.424 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:07.422+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:07.539 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:07.534+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:07.603 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:07.600+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:08.190 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:08.186+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:08.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:08.248+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:08.433 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:08.431+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:08.523 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:08.520+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:08.639 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:08.636+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:09.151 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:09.149+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:09.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:09.247+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:09.461 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:09.455+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:09.565 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:09.561+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:09.655 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:09.650+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:10.112 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:10.109+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:10.295 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:10.292+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:10.326 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:10.322+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:10.464 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:10.460+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:10.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:10.550+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:10.671 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:10.668+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:11.155 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:11.152+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:11.288 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:11.284+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:11.349 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:11.347+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:11.422 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:11.419+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:11.573 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:11.571+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:11.695 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:11.689+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:12.168 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:12.164+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:12.245 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:12.240+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:12.342 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:12.339+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:12.425 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:12.420+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:12.425 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:12.424+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:12.531 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:12.528+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:12.721 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:12.719+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:13.212 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:13.210+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:13.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:13.252+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:13.372 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:13.370+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:13.457 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:13.453+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:13.459 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:13.456+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:13.499 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:13.496+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:13.759 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:13.756+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:14.181 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:14.179+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:14.293 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:14.288+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:14.389 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:14.387+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:14.439 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:14.434+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:14.440 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:14.438+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:14.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:14.513+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:14.730 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:14.727+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:15.175 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:15.172+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:15.276 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:15.273+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:15.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:15.403+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:15.411 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:15.409+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:15.421 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:15.414+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:15.551 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:15.549+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:15.729 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:15.727+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:16.171 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:16.168+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:16.309 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:16.304+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:16.441 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:16.438+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:16.461 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:16.457+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:16.462 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:16.460+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:16.559 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:16.556+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:16.738 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:16.736+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:17.146 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:17.141+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:17.298 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:17.296+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:17.476 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:17.474+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:17.476 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:17.475+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:17.489 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:17.487+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:17.515 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:17.513+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:17.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:17.693+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:18.197 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:18.194+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:18.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:18.296+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:18.477 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:18.475+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:18.481 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:18.479+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:18.492 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:18.489+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:18.499 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:18.496+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:18.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:18.692+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:19.253 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:19.246+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:19.304 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:19.301+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:19.476 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:19.474+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:19.488 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:19.486+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:19.520 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:19.517+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:19.520 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:19.519+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:19.688 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:19.685+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:20.232 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:20.225+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:20.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:20.323+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:20.485 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:20.481+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:20.525 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:20.522+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:20.526 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:20.524+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:20.526 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:20.523+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:20.658 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:20.655+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:21.234 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:21.232+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:21.281 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:21.278+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:21.451 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:21.448+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:21.510 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:21.508+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:21.512 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:21.512+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:21.553 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:21.550+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:21.659 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:21.656+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:22.247 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:22.236+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:22.260 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:22.255+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:22.502 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:22.500+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:22.531 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:22.527+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:22.531 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:22.530+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:22.591 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:22.589+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:22.711 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:22.707+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:23.217 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:23.215+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:23.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:23.275+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:23.504 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:23.500+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:23.505 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:23.503+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:23.538 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:23.534+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:23.573 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:23.569+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:23.724 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:23.721+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:24.214 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:24.211+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:24.311 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:24.308+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:24.472 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:24.469+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:24.473 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:24.472+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:24.526 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:24.524+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:24.534 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:24.532+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:24.736 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:24.732+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:25.252 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:25.249+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:25.307 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:25.304+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:25.450 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:25.446+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:25.451 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:25.449+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:25.494 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:25.491+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:25.562 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:25.561+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:25.722 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:25.715+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:26.234 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:26.232+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:26.287 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:26.283+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:26.407 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:26.403+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:26.408 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:26.406+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:26.527 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:26.524+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:26.596 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:26.592+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:26.715 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:26.711+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:27.187 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:27.185+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:27.294 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:27.292+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:27.416 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:27.413+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:27.418 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:27.418+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:27.558 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:27.556+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:27.581 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:27.573+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:27.675 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:27.671+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:28.232 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:28.227+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:28.319 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:28.315+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:28.459 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:28.455+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:28.460 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:28.458+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:28.541 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:28.538+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:28.542 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:28.539+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:28.668 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:28.665+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:29.267 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:29.264+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:29.285 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:29.283+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:29.488 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:29.484+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:29.489 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:29.487+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:29.504 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:29.502+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:29.542 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:29.539+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:29.718 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:29.716+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:30.266 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:30.264+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:30.324 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:30.320+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:30.492 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:30.489+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:30.493 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:30.492+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:30.546 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:30.544+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:30.548 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:30.544+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:30.716 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:30.712+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:31.237 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:31.227+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:31.327 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:31.324+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:31.452 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:31.447+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:31.452 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:31.451+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:31.522 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:31.516+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:31.599 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:31.595+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:31.673 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:31.668+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:32.259 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:32.254+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:32.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:32.311+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:32.438 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:32.436+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:32.439 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:32.437+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:32.501 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:32.498+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:32.605 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:32.602+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:32.645 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:32.642+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:33.252 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:33.248+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:33.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:33.308+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:33.449 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:33.445+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:33.450 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:33.447+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:33.528 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:33.526+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:33.606 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:33.604+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:33.647 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:33.643+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:34.285 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:34.282+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:34.381 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:34.364+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:34.499 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:34.488+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:34.507 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:34.497+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:34.511 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:34.502+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:34.585 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:34.570+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:34.646 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:34.628+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:35.303 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:35.300+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:35.378 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:35.375+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:35.469 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:35.466+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:35.472 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:35.472+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:35.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:35.498+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:35.545 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:35.542+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:35.637 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:35.634+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:36.314 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:36.312+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:36.412 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:36.408+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:36.435 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:36.431+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:36.436 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:36.434+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:36.496 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:36.493+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:36.534 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:36.532+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:36.615 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:36.612+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:37.347 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:37.346+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:37.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:37.425+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:37.477 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:37.473+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:37.478 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:37.476+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:37.485 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:37.482+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:37.561 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:37.558+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:37.653 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:37.649+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:38.366 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:38.363+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:38.400 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:38.395+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:38.493 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:38.489+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:38.496 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:38.492+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:38.497 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:38.495+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:38.541 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:38.536+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:38.703 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:20:38.700+0000 1694f700 -1 osd.4 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.429623+0000 front 2019-12-12T00:19:22.442265+0000 (oldest deadline 2019-12-12T00:20:05.850750+0000) 2019-12-12T00:20:39.391 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:20:39.389+0000 1694f700 -1 osd.7 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:21.662597+0000 front 2019-12-12T00:19:21.552746+0000 (oldest deadline 2019-12-12T00:20:06.115979+0000) 2019-12-12T00:20:39.408 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:20:39.406+0000 1694f700 -1 osd.1 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:22.355052+0000 front 2019-12-12T00:19:22.353887+0000 (oldest deadline 2019-12-12T00:20:05.714565+0000) 2019-12-12T00:20:39.463 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:20:39.458+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 44sec e690 v690) 2019-12-12T00:20:39.478 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:39.475+0000 1694f700 -1 osd.6 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.106617+0000 front 2019-12-12T00:19:20.085816+0000 (oldest deadline 2019-12-12T00:20:05.940724+0000) 2019-12-12T00:20:39.479 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:39.477+0000 1694f700 -1 osd.6 690 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:39.503 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:20:39.501+0000 1694f700 -1 osd.3 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:20.306264+0000 front 2019-12-12T00:19:20.104699+0000 (oldest deadline 2019-12-12T00:20:05.360761+0000) 2019-12-12T00:20:39.554 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:20:39.551+0000 1694f700 -1 osd.2 690 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:19:24.015042+0000 front 2019-12-12T00:19:24.041304+0000 (oldest deadline 2019-12-12T00:20:09.911569+0000) 2019-12-12T00:20:40.520 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:40.515+0000 1694f700 -1 osd.6 691 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:20:41.555 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:20:41.542+0000 1694f700 -1 osd.6 692 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:123579 2.17s0 2:e877506b:::14af9c21-7f56-4de3-9a52-895b5fee19a0.4498.673_%e9%90%8b%eb%8c%8c%e3%9c%85%e5%98%88%eb%8c%8e%e8%94%8a%e1%a0%82%eb%8c%8d%e8%94%8b%ef%84%91%e1%a0%82%eb%8c%8b%e1%a0%84%ef%84%91%e1%a0%82:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e690) 2019-12-12T00:21:37.787 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:37.783+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:38.475 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:38.473+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:38.544 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:38.541+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:38.757 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:38.755+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:39.513 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:39.504+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:39.633 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:39.519+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:39.752 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:39.750+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:39.763 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:39.761+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:39.962 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:39.960+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:40.482 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:40.479+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:40.556 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:40.553+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:40.740 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:40.737+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:40.757 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:40.755+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:40.918 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:40.915+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:41.497 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:41.495+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:41.576 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:41.573+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:41.764 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:41.761+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:41.803 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:41.801+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:41.875 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:41.871+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:42.544 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:42.542+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:42.553 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:42.550+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:42.767 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:42.765+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:42.800 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:42.798+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:42.915 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:42.913+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:43.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:43.551+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:43.575 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:43.573+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:43.768 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:43.766+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:43.776 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:43.773+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:43.925 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:43.922+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:44.592 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:44.589+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:44.615 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:44.612+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:44.748 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:44.744+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:44.807 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:44.801+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:44.897 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:44.895+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:45.497 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:45.495+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:45.610 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:45.607+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:45.618 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:45.616+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:45.726 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:45.724+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:45.840 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:45.837+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:45.916 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:45.909+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:46.465 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:46.464+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:46.575 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:46.572+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:46.652 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:46.649+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:46.753 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:46.750+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:46.801 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:46.799+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:46.880 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:46.877+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:47.471 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:47.470+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:47.541 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:47.538+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:47.718 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:47.688+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:47.747 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:47.733+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:47.809 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:47.807+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:47.920 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:47.917+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:48.488 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:48.486+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:48.541 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:48.537+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:48.659 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:48.656+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:48.691 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:48.689+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:48.857 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:48.855+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:48.879 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:48.876+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:49.444 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:49.442+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:49.494 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:49.491+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:49.683 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:49.680+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:49.739 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:49.737+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:49.858 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:49.855+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:49.861 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:49.858+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:50.427 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:50.426+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:50.475 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:50.472+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:50.704 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:50.700+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:50.725 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:50.722+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:50.819 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:50.817+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:50.905 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:50.903+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:51.445 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:51.442+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:51.461 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:51.459+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:51.732 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:51.730+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:51.759 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:51.748+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:51.829 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:51.826+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:51.895 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:51.892+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:52.419 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:52.416+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:52.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:52.500+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:52.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:52.502+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:52.719 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:52.717+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:52.753 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:52.751+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:52.783 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:52.780+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:52.949 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:52.944+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:53.402 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:53.399+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:53.552 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:53.548+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:53.552 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:53.551+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:53.726 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:53.724+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:53.768 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:53.765+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:53.820 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:53.817+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:53.920 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:53.919+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:54.419 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:54.416+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:54.571 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:54.569+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:54.573 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:54.572+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:54.749 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:54.745+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:54.769 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:54.766+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:54.818 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:54.816+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:54.885 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:54.883+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:55.472 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:55.469+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:55.610 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:55.607+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:55.610 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:55.608+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:55.726 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:55.724+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:55.752 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:55.750+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:55.858 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:55.856+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:55.921 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:55.919+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:56.490 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:56.488+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:56.602 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:56.599+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:56.602 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:56.600+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:56.754 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:56.751+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:56.803 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:56.801+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:56.887 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:56.885+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:56.968 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:56.966+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:57.450 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:57.448+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:57.583 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:57.579+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:57.585 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:57.581+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:57.748 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:57.745+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:57.804 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:57.802+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:57.883 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:57.881+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:57.966 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:57.964+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:58.433 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:58.432+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:58.540 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:58.537+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:58.542 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:58.540+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:58.761 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:58.757+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:58.842 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:58.841+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:58.863 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:58.860+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:58.977 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:58.974+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:21:59.395 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:21:59.394+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:21:59.513 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:59.510+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:21:59.515 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:21:59.514+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:21:59.755 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:21:59.753+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:21:59.818 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:21:59.818+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:21:59.842 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:21:59.839+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:21:59.933 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:49:46.866 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:21:59.938 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:49:46.872 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:21:59.941 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:49:46.874 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:21:59.960 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:49:46.893 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:21:59.976 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:21:59.975+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:00.003 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:49:46.908 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:00.006 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:49:46.912 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:00.126 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:49:47.059 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:00.148 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:49:47.081 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:00.207 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:49:47.113 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:00.451 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:00.449+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:00.489 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:00.486+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:00.490 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:00.487+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:00.725 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:00.722+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:00.845 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:00.843+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:00.846 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:00.843+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:00.938 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:00.937+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:01.457 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:01.452+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:01.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:01.451+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:01.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:01.452+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:01.752 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:01.746+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:01.809 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:01.807+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:01.846 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:01.842+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:01.959 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:01.951+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:02.424 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:02.422+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:02.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:02.499+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:02.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:02.502+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:02.791 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:02.789+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:02.858 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:02.856+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:02.897 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:02.895+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:02.970 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:02.968+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:03.414 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:03.405+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:03.478 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:03.476+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:03.479 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:03.478+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:03.805 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:03.803+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:03.855 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:03.847+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:03.865 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:03.862+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:04.019 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:04.017+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:04.426 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:04.425+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:04.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:04.501+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:04.504 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:04.503+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:04.830 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:04.828+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:04.883 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:04.880+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:04.883 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:04.881+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:05.037 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:05.035+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:05.446 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:05.444+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:05.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:05.533+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:05.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:05.534+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:05.883 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:05.880+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:05.913 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:05.910+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:05.937 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:05.932+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:06.019 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:06.017+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:06.489 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:06.486+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:06.490 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:06.487+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:06.500 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:06.497+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:06.890 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:06.887+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:06.900 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:06.897+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:06.907 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:06.898+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:07.030 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:07.027+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:07.478 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:07.475+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:07.519 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:07.516+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:07.519 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:07.517+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:07.887 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:07.885+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:07.923 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:07.921+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:07.951 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:07.948+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:08.004 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:07.999+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:08.459 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:08.456+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:08.526 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:08.522+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:08.527 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:08.524+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:08.899 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:08.896+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:08.936 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:08.933+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:08.961 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:08.957+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:08.963 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:08.961+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:09.490 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:09.488+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:09.518 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:09.514+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:09.519 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:09.516+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:09.895 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:22:09.890+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e696 v696) 2019-12-12T00:22:09.908 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:09.904+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:09.951 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:09.948+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:09.971 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:09.969+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:09.989 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:09.986+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:10.500 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:10.498+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:10.517 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:10.514+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:10.518 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:10.515+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:10.865 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:10.861+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:10.916 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:10.912+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:10.932 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:10.930+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:11.034 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:11.031+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:11.492 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:11.490+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:11.528 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:11.524+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:11.529 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:11.526+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:11.824 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:11.820+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:11.914 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:11.911+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:11.988 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:11.985+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:12.043 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:12.040+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:12.493 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:12.490+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:12.495 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:12.493+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:12.525 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:12.523+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:12.837 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:12.833+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:12.917 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:12.912+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:13.000 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:12.996+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:13.018 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:13.016+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:13.452 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:13.449+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:13.453 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:13.451+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:13.552 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:13.549+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:13.810 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:13.806+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:13.926 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:13.922+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:13.991 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:13.988+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:14.005 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:14.003+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:14.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:14.409+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:14.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:14.410+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:14.525 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:14.523+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:14.816 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:14.812+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:14.897 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:14.894+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:14.913 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:22:14.909+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e696 v696) 2019-12-12T00:22:14.985 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:14.983+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:15.008 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:15.006+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:15.398 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:15.393+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:15.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:15.396+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:15.570 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:15.568+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:15.866 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:15.863+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:15.894 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:15.892+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:15.964 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:15.961+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:15.987 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:15.985+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:16.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:16.411+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:16.414 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:16.412+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:16.592 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:16.589+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:16.849 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:16.845+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:16.922 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:16.918+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:16.994 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:16.991+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:17.021 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:17.018+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:17.396 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:17.393+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:17.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:17.396+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:17.561 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:17.558+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:17.875 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:17.871+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:17.877 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:17.871+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:17.984 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:17.978+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:18.022 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:18.020+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:18.399 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:18.397+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:18.401 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:18.399+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:18.578 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:18.575+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:18.875 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:18.872+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:18.888 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:18.885+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:19.013 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:19.010+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:19.062 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:19.059+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:19.451 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:19.449+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:19.457 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:19.454+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:19.592 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:19.589+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:19.830 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:19.828+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:19.930 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:19.927+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:19.931 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:22:19.930+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e696 v696) 2019-12-12T00:22:20.014 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:20.013+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:20.099 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:20.097+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:20.508 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:20.505+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:20.509 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:20.507+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:20.584 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:20.581+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:20.870 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:20.866+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:20.887 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:20.885+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:21.022 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:21.019+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:21.152 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:21.150+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:21.521 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:21.519+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:21.522 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:21.520+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:21.634 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:21.632+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:21.859 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:21.857+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:21.909 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:21.906+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:22.029 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:22.026+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:22.161 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:22.158+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:22.571 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:22.566+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:22.572 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:22.568+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:22.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:22.683+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:22.864 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:22.861+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:22.945 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:22.941+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:23.025 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:23.023+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:23.150 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:23.147+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:23.588 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:23.586+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:23.589 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:23.588+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:23.720 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:23.717+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:23.905 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:23.903+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:23.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:23.941+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:24.079 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:24.075+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:24.191 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:24.188+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:24.558 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:24.555+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:24.559 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:24.557+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:24.714 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:24.711+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:24.898 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:24.895+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:24.913 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:24.911+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:24.953 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:22:24.948+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e696 v696) 2019-12-12T00:22:25.094 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:25.092+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:25.170 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:25.167+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:25.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:25.536+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:25.539 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:25.537+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:25.765 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:25.761+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:25.928 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:25.926+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:25.953 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:25.949+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:26.111 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:26.109+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:26.146 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:26.143+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:26.550 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:26.547+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:26.553 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:26.551+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:26.738 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:26.736+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:26.941 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:26.932+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:26.997 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:26.993+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:27.077 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:27.074+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:27.118 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:27.115+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:27.605 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:27.602+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:27.611 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:27.608+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:27.734 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:27.731+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:27.946 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:27.943+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:28.027 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:28.022+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:28.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:28.082+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:28.125 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:28.121+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:28.643 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:28.640+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:28.646 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:28.644+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:28.701 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:28.698+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:28.898 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:28.896+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:29.067 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:29.064+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:29.096 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:29.093+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:29.124 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:22:29.121+0000 1694f700 -1 osd.6 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.857619+0000 front 2019-12-12T00:20:53.909985+0000 (oldest deadline 2019-12-12T00:21:39.041329+0000) 2019-12-12T00:22:29.691 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:29.689+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:29.694 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:29.692+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:29.736 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:29.733+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:29.863 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:22:29.861+0000 1694f700 -1 osd.2 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:55.466407+0000 front 2019-12-12T00:20:55.509064+0000 (oldest deadline 2019-12-12T00:21:39.293465+0000) 2019-12-12T00:22:30.025 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:22:30.015+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 43sec e696 v696) 2019-12-12T00:22:30.052 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:22:30.049+0000 1694f700 -1 osd.1 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:54.439832+0000 front 2019-12-12T00:20:54.524766+0000 (oldest deadline 2019-12-12T00:21:37.323302+0000) 2019-12-12T00:22:30.104 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:22:30.091+0000 1694f700 -1 osd.7 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:56.827548+0000 front 2019-12-12T00:20:56.846874+0000 (oldest deadline 2019-12-12T00:21:37.893554+0000) 2019-12-12T00:22:30.728 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:30.722+0000 1694f700 -1 osd.0 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 ever on either front or back, first ping sent 2019-12-12T00:21:11.495300+0000 (oldest deadline 2019-12-12T00:21:51.495300+0000) 2019-12-12T00:22:30.729 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:22:30.723+0000 1694f700 -1 osd.0 696 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4498.0:88594 7.f 7.152d324f (undecoded) ondisk+retry+read+known_if_redirected e694) 2019-12-12T00:22:30.734 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:22:30.733+0000 1694f700 -1 osd.4 696 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:20:53.623313+0000 front 2019-12-12T00:20:53.626200+0000 (oldest deadline 2019-12-12T00:21:38.300182+0000) 2019-12-12T00:22:41.528 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:50:28.461 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.536 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:50:28.469 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.545 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:50:28.478 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.552 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:50:28.485 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.560 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:50:28.466 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.586 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:50:28.491 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.633 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:50:28.566 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.645 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:50:28.578 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:22:41.715 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:50:28.620 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:23:59.709 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:23:59.704+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:00.675 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:00.671+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:01.522 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:01.519+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:01.617 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:01.614+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:01.709 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:01.705+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:02.574 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:02.570+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:02.581 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:02.578+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:02.602 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:02.598+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:02.698 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:02.693+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:02.698 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:02.694+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:03.559 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:03.552+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:03.589 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:03.587+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:03.611 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:03.608+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:03.636 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:03.633+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:03.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:03.695+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:03.737 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:03.733+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:04.517 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:04.513+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:04.599 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:04.594+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:04.599 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:04.595+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:04.643 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:04.639+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:04.697 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:04.694+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:04.720 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:04.716+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:05.476 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:05.473+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:05.570 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:05.567+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:05.574 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:05.571+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:05.676 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:05.672+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:05.681 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:05.678+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:05.684 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:05.681+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:06.512 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:06.508+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:06.549 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:06.545+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:06.578 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:06.576+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:06.649 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:06.646+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:06.712 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:06.708+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:06.717 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:06.713+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:07.536 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:07.532+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:07.554 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:07.548+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:07.587 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:07.582+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:07.635 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:07.630+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:07.705 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:07.696+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:07.738 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:07.736+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:08.501 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:08.498+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:08.551 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:08.548+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:08.558 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:08.555+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:08.638 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:08.635+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:08.658 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:08.654+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:08.759 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:08.753+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:09.480 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:09.477+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:09.547 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:09.543+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:09.548 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:09.545+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:09.653 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:09.650+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:09.711 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:09.707+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:09.762 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:09.759+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:10.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:10.455+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:10.517 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:10.513+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:10.594 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:10.591+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:10.657 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:10.653+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:10.736 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:10.732+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:10.745 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:10.739+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:11.464 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:11.462+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:11.506 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:11.503+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:11.655 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:11.634+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:11.678 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:11.668+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:11.713 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:11.711+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:11.762 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:11.758+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:12.470 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:12.468+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:12.500 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:12.494+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:12.700 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:12.697+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:12.700 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:12.656+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:12.726 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:12.723+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:12.764 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:12.760+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:13.463 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:13.460+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:13.471 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:13.468+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:13.728 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:13.725+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:13.740 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:13.738+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:13.776 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:13.773+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:13.802 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:13.799+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:14.438 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:14.433+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:14.475 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:14.471+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:14.749 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:14.744+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:14.765 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:14.762+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:14.778 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:14.775+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:14.783 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:14.779+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:15.482 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:15.479+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:15.483 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:15.481+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:15.770 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:15.767+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:15.784 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:15.780+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:15.796 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:15.792+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:15.796 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:15.793+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:16.439 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:16.435+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:16.471 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:16.469+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:16.793 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:16.785+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:16.802 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:16.798+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:16.804 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:16.801+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:16.848 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:16.843+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:17.404 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:17.401+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:17.430 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:17.427+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:17.747 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:17.744+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:17.788 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:17.784+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:17.817 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:17.804+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:17.891 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:17.886+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:18.356 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:18.353+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:18.481 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:18.478+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:18.755 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:18.751+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:18.756 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:18.753+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:18.814 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:18.811+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:18.853 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:18.850+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:19.315 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:19.312+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:19.516 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:19.512+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:19.776 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:19.773+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:19.788 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:19.785+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:19.837 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:19.833+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:19.862 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:19.858+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:20.296 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:20.292+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:20.556 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:20.552+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:20.762 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:20.759+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:20.765 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:20.750+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:20.834 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:20.830+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:20.875 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:20.872+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:21.305 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:21.301+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:21.548 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:21.544+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:21.738 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:21.734+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:21.804 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:21.800+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:21.827 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:21.822+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:21.927 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:21.923+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:22.319 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:22.314+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:22.557 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:22.552+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:22.698 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:22.695+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:22.789 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:22.784+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:22.794 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:22.791+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:22.943 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:22.940+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:23.364 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:23.361+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:23.511 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:23.508+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:23.661 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:23.657+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:23.765 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:23.761+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:23.827 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:23.824+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:23.908 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:23.904+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:24.353 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:24.341+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:24.487 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:24.483+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:24.670 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:24.667+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:24.790 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:24.784+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:24.813 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:24.810+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:24.912 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:24.871+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:25.314 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:25.309+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:25.478 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:25.474+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:25.680 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:25.677+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:25.841 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:25.836+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:25.862 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:25.858+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:25.884 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:25.881+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:26.274 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:26.270+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:26.497 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:26.493+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:26.707 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:26.703+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:26.848 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:26.839+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:26.857 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:26.853+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:26.858 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:26.855+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:27.286 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:27.282+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:27.505 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:27.502+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:27.698 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:27.695+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:27.859 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:27.854+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:27.887 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:27.883+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:27.892 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:27.889+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:28.286 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:28.283+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:28.464 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:28.461+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:28.743 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:28.740+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:28.847 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:28.843+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:28.881 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:28.875+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:28.928 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:28.924+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:29.340 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:29.336+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:29.442 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:29.438+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:29.786 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:29.782+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:29.842 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:29.839+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:29.896 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:29.893+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:29.913 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:29.910+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:30.309 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:30.305+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:30.411 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:30.407+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:30.803 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:30.799+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:30.868 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:30.865+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:30.949 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:30.917+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:30.957 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:30.954+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:31.272 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:31.267+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:31.392 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:31.390+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:31.851 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:31.847+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:31.886 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:31.883+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:31.952 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:31.948+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:32.010 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:32.006+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:32.312 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:32.308+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:32.413 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:32.410+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:32.823 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:32.820+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:32.907 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:32.904+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:32.945 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:32.936+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:32.995 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:32.992+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:33.317 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:33.313+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:33.443 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:33.441+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:33.855 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:33.850+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:33.878 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:33.876+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:33.925 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:33.923+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:33.985 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:33.982+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:34.292 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:34.288+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:34.454 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:34.451+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:24:34.865 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:24:34.861+0000 1694f700 -1 osd.6 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:18.018097+0000 front 2019-12-12T00:23:18.466314+0000 (oldest deadline 2019-12-12T00:24:02.705915+0000) 2019-12-12T00:24:34.890 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:24:34.887+0000 1694f700 -1 osd.4 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:13.795298+0000 front 2019-12-12T00:23:13.954456+0000 (oldest deadline 2019-12-12T00:23:59.549195+0000) 2019-12-12T00:24:34.906 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:24:34.903+0000 1694f700 -1 osd.7 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.531682+0000 front 2019-12-12T00:23:17.447153+0000 (oldest deadline 2019-12-12T00:24:02.494876+0000) 2019-12-12T00:24:34.996 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:24:34.993+0000 1694f700 -1 osd.2 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:17.981155+0000 front 2019-12-12T00:23:17.878592+0000 (oldest deadline 2019-12-12T00:24:02.514558+0000) 2019-12-12T00:24:35.248 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:24:35.244+0000 1694f700 -1 osd.3 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:16.820065+0000 front 2019-12-12T00:23:16.794677+0000 (oldest deadline 2019-12-12T00:24:00.770506+0000) 2019-12-12T00:24:35.439 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:24:35.431+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.21.15.79:6826/10221,v1:172.21.15.79:6827/10221] for 47sec e703 v703) 2019-12-12T00:24:35.439 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:24:35.436+0000 1694f700 -1 osd.1 703 heartbeat_check: no reply from 172.21.15.79:6830 osd.0 since back 2019-12-12T00:23:15.723091+0000 front 2019-12-12T00:23:15.440746+0000 (oldest deadline 2019-12-12T00:24:00.730731+0000) 2019-12-12T00:25:08.300 INFO:tasks.ceph.osd.0.smithi079.stderr:*** Caught signal (Segmentation fault) ** 2019-12-12T00:25:08.300 INFO:tasks.ceph.osd.0.smithi079.stderr: in thread 11caf700 thread_name:msgr-worker-1 2019-12-12T00:25:08.353 INFO:tasks.ceph.osd.0.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:25:08.354 INFO:tasks.ceph.osd.0.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:25:08.354 INFO:tasks.ceph.osd.0.smithi079.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-12T00:25:08.354 INFO:tasks.ceph.osd.0.smithi079.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-12T00:25:08.354 INFO:tasks.ceph.osd.0.smithi079.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-12T00:25:08.354 INFO:tasks.ceph.osd.0.smithi079.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-12T00:25:08.354 INFO:tasks.ceph.osd.0.smithi079.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-12T00:25:08.355 INFO:tasks.ceph.osd.0.smithi079.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-12T00:25:08.355 INFO:tasks.ceph.osd.0.smithi079.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-12T00:25:08.355 INFO:tasks.ceph.osd.0.smithi079.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-12T00:25:08.355 INFO:tasks.ceph.osd.0.smithi079.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-12T00:25:08.355 INFO:tasks.ceph.osd.0.smithi079.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-12T00:25:08.355 INFO:tasks.ceph.osd.0.smithi079.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-12T00:25:08.355 INFO:tasks.ceph.osd.0.smithi079.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:25:08.382 INFO:tasks.ceph.osd.0.smithi079.stderr:2019-12-12T00:25:08.375+0000 11caf700 -1 *** Caught signal (Segmentation fault) ** 2019-12-12T00:25:08.382 INFO:tasks.ceph.osd.0.smithi079.stderr: in thread 11caf700 thread_name:msgr-worker-1 2019-12-12T00:25:08.382 INFO:tasks.ceph.osd.0.smithi079.stderr: 2019-12-12T00:25:08.382 INFO:tasks.ceph.osd.0.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:25:08.382 INFO:tasks.ceph.osd.0.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:25:08.382 INFO:tasks.ceph.osd.0.smithi079.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-12T00:25:08.383 INFO:tasks.ceph.osd.0.smithi079.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-12T00:25:08.383 INFO:tasks.ceph.osd.0.smithi079.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-12T00:25:08.383 INFO:tasks.ceph.osd.0.smithi079.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-12T00:25:08.383 INFO:tasks.ceph.osd.0.smithi079.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-12T00:25:08.383 INFO:tasks.ceph.osd.0.smithi079.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-12T00:25:08.383 INFO:tasks.ceph.osd.0.smithi079.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-12T00:25:08.383 INFO:tasks.ceph.osd.0.smithi079.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-12T00:25:08.384 INFO:tasks.ceph.osd.0.smithi079.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-12T00:25:08.384 INFO:tasks.ceph.osd.0.smithi079.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-12T00:25:08.384 INFO:tasks.ceph.osd.0.smithi079.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-12T00:25:08.384 INFO:tasks.ceph.osd.0.smithi079.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:25:08.384 INFO:tasks.ceph.osd.0.smithi079.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-12T00:25:08.385 INFO:tasks.ceph.osd.0.smithi079.stderr: 2019-12-12T00:25:08.822 INFO:tasks.ceph.osd.0.smithi079.stderr: -11> 2019-12-12T00:25:08.375+0000 11caf700 -1 *** Caught signal (Segmentation fault) ** 2019-12-12T00:25:08.822 INFO:tasks.ceph.osd.0.smithi079.stderr: in thread 11caf700 thread_name:msgr-worker-1 2019-12-12T00:25:08.822 INFO:tasks.ceph.osd.0.smithi079.stderr: 2019-12-12T00:25:08.822 INFO:tasks.ceph.osd.0.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:25:08.822 INFO:tasks.ceph.osd.0.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:25:08.823 INFO:tasks.ceph.osd.0.smithi079.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-12T00:25:08.823 INFO:tasks.ceph.osd.0.smithi079.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-12T00:25:08.823 INFO:tasks.ceph.osd.0.smithi079.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-12T00:25:08.823 INFO:tasks.ceph.osd.0.smithi079.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-12T00:25:08.823 INFO:tasks.ceph.osd.0.smithi079.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-12T00:25:08.823 INFO:tasks.ceph.osd.0.smithi079.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-12T00:25:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-12T00:25:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-12T00:25:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-12T00:25:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-12T00:25:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-12T00:25:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:25:08.824 INFO:tasks.ceph.osd.0.smithi079.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-12T00:25:08.825 INFO:tasks.ceph.osd.0.smithi079.stderr: 2019-12-12T00:25:09.197 INFO:tasks.ceph.osd.0.smithi079.stderr: -1012> 2019-12-12T00:25:08.375+0000 11caf700 -1 *** Caught signal (Segmentation fault) ** 2019-12-12T00:25:09.197 INFO:tasks.ceph.osd.0.smithi079.stderr: in thread 11caf700 thread_name:msgr-worker-1 2019-12-12T00:25:09.197 INFO:tasks.ceph.osd.0.smithi079.stderr: 2019-12-12T00:25:09.197 INFO:tasks.ceph.osd.0.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:25:09.197 INFO:tasks.ceph.osd.0.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:25:09.197 INFO:tasks.ceph.osd.0.smithi079.stderr: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0xe8a9b1] 2019-12-12T00:25:09.198 INFO:tasks.ceph.osd.0.smithi079.stderr: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x100d020] 2019-12-12T00:25:09.198 INFO:tasks.ceph.osd.0.smithi079.stderr: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x1019dc3] 2019-12-12T00:25:09.198 INFO:tasks.ceph.osd.0.smithi079.stderr: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x101a040] 2019-12-12T00:25:09.198 INFO:tasks.ceph.osd.0.smithi079.stderr: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x101a59d] 2019-12-12T00:25:09.198 INFO:tasks.ceph.osd.0.smithi079.stderr: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x1003224] 2019-12-12T00:25:09.198 INFO:tasks.ceph.osd.0.smithi079.stderr: 8: (AsyncConnection::process()+0x794) [0xfdd874] 2019-12-12T00:25:09.199 INFO:tasks.ceph.osd.0.smithi079.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0xe3dd55] 2019-12-12T00:25:09.199 INFO:tasks.ceph.osd.0.smithi079.stderr: 10: (()+0xd3b767) [0xe43767] 2019-12-12T00:25:09.199 INFO:tasks.ceph.osd.0.smithi079.stderr: 11: (()+0x1214fdf) [0x131cfdf] 2019-12-12T00:25:09.199 INFO:tasks.ceph.osd.0.smithi079.stderr: 12: (()+0x7e65) [0xc096e65] 2019-12-12T00:25:09.199 INFO:tasks.ceph.osd.0.smithi079.stderr: 13: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:25:09.199 INFO:tasks.ceph.osd.0.smithi079.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-12T00:25:09.199 INFO:tasks.ceph.osd.0.smithi079.stderr: 2019-12-12T00:25:13.304 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:25:13.155+0000 31f87700 -1 osd.3 707 build_incremental_map_msg missing incremental map 690 2019-12-12T00:25:14.325 INFO:tasks.ceph.osd.0.smithi079.stderr:daemon-helper: command crashed with signal 11 2019-12-12T00:25:14.764 INFO:teuthology.orchestra.run.smithi079.stderr:test_slo_etag_is_hash_of_etags (test.functional.tests.TestSlo) ... ok 2019-12-12T00:25:16.744 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:16.738+0000 35f8f700 -1 osd.2 707 build_incremental_map_msg missing incremental map 669 2019-12-12T00:25:17.488 INFO:teuthology.orchestra.run.smithi079.stderr:test_slo_get_simple_manifest (test.functional.tests.TestSlo) ... ok 2019-12-12T00:25:19.904 INFO:teuthology.orchestra.run.smithi079.stderr:test_slo_ranged_get (test.functional.tests.TestSlo) ... ok 2019-12-12T00:25:33.926 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:33.923+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:33.980 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:33.976+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:34.707 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:34.703+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:34.902 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:34.899+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:35.009 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:35.006+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:35.755 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:35.752+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:35.910 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:35.906+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:35.994 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:35.991+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:36.794 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:36.791+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:36.956 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:36.953+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:37.029 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:37.026+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:37.164 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:37.158+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:37.761 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:37.757+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:37.941 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:37.937+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:37.974 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:37.970+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:38.011 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:38.008+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:38.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:38.122+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:38.765 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:38.762+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:38.978 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:38.974+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:39.013 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:39.010+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:39.013 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:39.007+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:39.163 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:39.160+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:39.749 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:39.743+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:39.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:39.975+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:39.982 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:39.978+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:40.009 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:40.006+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:40.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:40.164+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:40.729 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:40.726+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:40.978 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:40.976+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:40.992 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:40.990+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:41.011 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:41.008+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:41.152 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:41.150+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:41.722 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:41.719+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:41.963 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:41.961+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:42.015 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:42.009+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:42.038 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:42.034+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:42.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:42.149+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:42.755 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:42.752+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:42.927 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:42.921+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:43.053 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:43.048+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:43.069 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:43.065+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:43.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:43.166+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:43.777 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:43.774+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:43.946 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:43.942+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:44.105 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:44.103+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:44.123 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:44.120+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:44.124 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:44.123+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:44.825 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:44.822+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:44.981 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:44.970+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:45.095 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:45.093+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:45.100 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:45.097+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:45.117 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:45.115+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:45.838 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:45.834+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:45.964 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:45.962+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:46.098 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:46.096+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:46.110 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:46.102+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:46.150 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:46.149+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:46.843 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:46.840+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:46.974 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:46.971+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:47.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:47.103+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:47.120 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:47.118+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:47.196 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:47.193+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:47.861 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:47.856+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:47.996 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:47.992+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:48.106 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:48.102+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:48.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:48.101+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:48.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:48.200+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:48.841 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:48.838+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:48.983 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:48.982+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:49.099 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:49.094+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:49.141 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:49.136+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:49.227 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:49.225+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:49.875 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:49.870+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:49.985 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:49.983+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:50.111 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:50.106+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:50.160 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:50.157+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:50.200 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:50.198+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:50.876 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:50.873+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:51.025 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:51.022+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:51.090 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:51.086+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:51.125 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:51.122+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:51.246 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:51.244+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:51.843 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:51.831+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:52.031 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:52.028+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:52.088 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:52.083+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:52.088 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:52.085+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:52.113 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:52.109+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:52.210 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:52.207+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:52.828 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:52.823+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:53.066 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:53.063+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:53.133 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:53.129+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:53.136 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:53.133+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:53.139 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:53.136+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:53.224 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:53.220+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:53.847 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:53.844+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:54.022 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:54.019+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:54.095 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:54.092+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:54.097 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:54.095+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:54.170 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:54.168+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:54.274 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:54.272+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:54.863 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:54.854+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:55.052 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:55.049+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:55.053 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:55.050+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:55.061 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:55.059+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:55.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:55.205+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:55.250 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:55.247+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:55.829 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:55.826+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:56.054 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:53:42.987 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.084 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:56.080+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:56.088 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:53:43.020 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.095 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:56.092+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:56.096 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:56.094+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:56.147 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:53:43.079 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.149 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:53:43.081 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.156 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:53:43.061 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.160 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:53:43.065 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.213 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:56.209+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:56.218 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:56.215+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:56.306 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:53:43.239 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.372 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:53:43.277 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.378 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:53:43.310 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:25:56.857 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:56.844+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:57.043 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:57.040+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:57.050 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:57.047+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:57.050 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:57.048+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:57.189 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:57.187+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:57.263 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:57.261+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:57.820 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:57.816+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:58.052 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:58.049+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:58.067 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:58.063+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:58.070 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:58.067+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:58.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:58.197+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:58.265 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:58.262+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:58.871 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:58.867+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:25:59.037 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:25:59.033+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:25:59.115 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:59.108+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:25:59.116 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:25:59.111+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:25:59.236 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:25:59.233+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:25:59.312 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:25:59.309+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:25:59.915 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:25:59.913+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:26:00.083 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:26:00.081+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:26:00.141 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:00.138+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:26:00.142 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:00.140+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:00.268 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:26:00.265+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:26:00.285 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:26:00.282+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:26:00.883 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:26:00.880+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:26:01.075 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:26:01.072+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:26:01.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:01.105+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:26:01.109 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:01.107+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:01.246 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:26:01.242+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:26:01.278 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:26:01.275+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:26:01.904 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:26:01.900+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:26:02.094 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:02.090+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:26:02.094 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:02.092+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:02.113 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:26:02.110+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:26:02.282 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:26:02.275+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:26:02.310 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:26:02.308+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:26:02.896 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:26:02.893+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:26:03.094 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:03.091+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:26:03.096 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:03.094+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:03.159 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:26:03.156+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:26:03.295 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:26:03.291+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:26:03.309 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:26:03.304+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:26:03.938 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:26:03.934+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:26:04.120 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:26:04.116+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:26:04.127 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:04.122+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:26:04.128 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:04.123+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:04.288 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:26:04.285+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:26:04.289 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:26:04.286+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:26:04.943 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:26:04.937+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:26:05.115 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:26:05.112+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:26:05.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:05.140+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:26:05.145 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:05.142+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:05.253 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:26:05.250+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:26:05.258 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:26:05.255+0000 1694f700 -1 osd.7 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:50.297565+0000 front 2019-12-12T00:24:50.298370+0000 (oldest deadline 2019-12-12T00:25:33.676090+0000) 2019-12-12T00:26:05.916 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:26:05.913+0000 1694f700 -1 osd.1 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.224905+0000 front 2019-12-12T00:24:51.209531+0000 (oldest deadline 2019-12-12T00:25:34.084319+0000) 2019-12-12T00:26:06.087 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:26:06.084+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e707 v707) 2019-12-12T00:26:06.109 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:26:06.104+0000 1694f700 -1 osd.6 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:47.937637+0000 front 2019-12-12T00:24:47.949935+0000 (oldest deadline 2019-12-12T00:25:33.805595+0000) 2019-12-12T00:26:06.124 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:06.121+0000 1694f700 -1 osd.2 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:51.718206+0000 front 2019-12-12T00:24:51.691619+0000 (oldest deadline 2019-12-12T00:25:36.964900+0000) 2019-12-12T00:26:06.128 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:06.123+0000 1694f700 -1 osd.2 707 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:06.250 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:26:06.247+0000 1694f700 -1 osd.4 707 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:24:52.264966+0000 front 2019-12-12T00:24:52.297391+0000 (oldest deadline 2019-12-12T00:25:36.304556+0000) 2019-12-12T00:26:07.417 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:07.234+0000 1694f700 -1 osd.2 708 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:08.456 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:26:08.348+0000 1694f700 -1 osd.2 709 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124523 6.f 6:f3bf49f5:root::.bucket.meta.:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e707) 2019-12-12T00:26:42.413 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:54:29.344 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.421 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:54:29.350 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.437 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:54:29.365 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.440 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:54:29.372 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.448 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:54:29.351 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.464 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:54:29.366 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.513 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:54:29.444 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.513 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:54:29.445 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:42.601 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:54:29.501 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:26:50.779 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:26:50.768+0000 36790700 -1 osd.3 713 build_incremental_map_msg missing incremental map 713 2019-12-12T00:26:50.805 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:26:50.799+0000 36790700 -1 osd.3 713 build_incremental_map_msg missing incremental map 713 2019-12-12T00:26:50.854 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:26:50.843+0000 36f91700 -1 osd.3 713 build_incremental_map_msg missing incremental map 713 2019-12-12T00:26:51.799 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:26:51.784+0000 38f95700 -1 osd.3 713 build_incremental_map_msg missing incremental map 713 2019-12-12T00:26:53.786 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:26:53.658+0000 31786700 -1 osd.3 713 build_incremental_map_msg missing incremental map 713 2019-12-12T00:27:36.038 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:36.036+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:37.057 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:37.054+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:38.081 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:38.079+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:39.102 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:39.099+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:40.094 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:40.091+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:41.065 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:41.063+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:42.071 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:42.069+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:43.086 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:43.083+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:44.102 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:44.099+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:45.142 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:45.140+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:46.144 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:46.139+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:47.181 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:47.179+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:48.032 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:48.027+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:48.174 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:48.172+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:49.044 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:49.041+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:49.196 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:49.193+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:49.196 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:49.194+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:49.283 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:49.280+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:49.914 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:49.910+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:50.045 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:50.041+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:50.162 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:50.158+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:50.163 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:50.160+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:50.249 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:50.246+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:50.876 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:50.869+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:50.916 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:50.908+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:51.089 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:51.061+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:51.211 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:51.198+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:51.212 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:51.202+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:51.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:51.229+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:51.845 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:51.837+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:51.929 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:51.926+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:52.091 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:52.088+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:52.244 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:52.240+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:52.255 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:52.251+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:52.255 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:52.253+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:52.865 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:52.862+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:52.958 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:52.953+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:53.115 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:53.111+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:53.236 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:53.232+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:53.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:53.292+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:53.297 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:53.294+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:53.878 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:53.873+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:53.912 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:53.910+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:54.114 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:54.111+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:54.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:54.241+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:54.280 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:54.276+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:54.280 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:54.277+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:54.875 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:54.871+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:54.914 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:54.910+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:55.086 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:55.083+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:55.212 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:55.209+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:55.299 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:55.295+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:55.300 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:55.297+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:55.857 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:55.851+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:55.933 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:55.930+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:56.057 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:56.052+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:56.237 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:56.234+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:56.347 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:56.342+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:56.348 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:56.344+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:56.857 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:56.853+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:56.941 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:56.938+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:57.009 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:57.006+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:57.272 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:57.266+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:57.398 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:57.394+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:57.399 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:57.396+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:57.847 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:57.833+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:57.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:57.932+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:58.026 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:58.023+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:58.228 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:58.224+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:58.442 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:58.437+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:58.443 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:58.440+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:58.811 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:58.806+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:58.897 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:58.892+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:27:58.985 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:27:58.980+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:27:59.203 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:27:59.199+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:27:59.463 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:59.459+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:27:59.465 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:27:59.461+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:27:59.783 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:27:59.781+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:27:59.922 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:27:59.918+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:00.036 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:00.026+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:00.237 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:00.234+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:00.483 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:00.477+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:00.483 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:00.479+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:00.822 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:00.819+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:00.957 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:00.950+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:01.005 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:00.999+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:01.208 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:01.205+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:01.447 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:01.442+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:01.447 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:01.443+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:01.832 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:01.826+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:01.946 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:01.943+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:01.993 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:01.988+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:02.223 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:02.220+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:02.496 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:02.491+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:02.497 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:02.493+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:02.833 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:02.826+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:02.971 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:02.963+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:02.993 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:02.988+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:03.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:03.247+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:03.544 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:03.540+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:03.545 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:03.541+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:03.809 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:03.801+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:03.951 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:03.947+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:04.033 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:04.029+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:04.212 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:04.208+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:04.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:04.579+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:04.595 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:04.583+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:04.836 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:04.833+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:04.980 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:04.976+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:05.059 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:05.057+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:05.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:05.189+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:05.577 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:05.574+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:05.578 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:05.576+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:05.879 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:05.866+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:05.959 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:05.955+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:06.070 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:06.067+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:06.171 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:06.168+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:06.597 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:06.593+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:06.598 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:06.595+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:06.923 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:06.920+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:06.938 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:06.932+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:07.112 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:07.109+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:07.191 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:07.188+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:07.594 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:07.589+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:07.596 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:07.591+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:07.931 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:07.928+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:07.969 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:07.965+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:08.085 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:08.081+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:08.206 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:08.203+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:08.559 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:08.556+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:08.562 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:08.559+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:08.959 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:08.956+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:08.973 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:08.970+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:09.049 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:09.046+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:09.254 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:09.250+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:09.569 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:09.560+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:09.569 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:09.562+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:09.937 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:09.935+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:09.976 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:09.974+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:10.017 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:10.008+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:10.291 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:10.288+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:10.549 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:10.546+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:10.549 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:10.547+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:10.905 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:10.900+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:10.998 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:10.994+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:11.064 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:11.060+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:11.286 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:11.283+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:11.561 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:11.558+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:11.561 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:11.559+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:11.951 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:11.947+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:12.022 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:12.019+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:12.077 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:12.073+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:12.342 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:12.334+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:12.553 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:12.548+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:12.554 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:12.549+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:12.974 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:12.970+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:13.042 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:13.039+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:13.125 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:13.122+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:13.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:13.311+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:13.553 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:13.549+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:13.553 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:13.551+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:13.950 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:13.946+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:14.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:14.063+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:14.087 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:14.084+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:14.279 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:14.276+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:14.578 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:14.574+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:14.579 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:14.576+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:14.933 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:14.930+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:15.046 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:15.043+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:15.059 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:15.055+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:15.333 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:15.331+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:15.538 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:15.533+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:15.539 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:15.535+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:15.928 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:15.926+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:16.034 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:16.032+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:16.059 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:16.056+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:16.349 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:16.347+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:16.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:16.566+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:16.581 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:16.567+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:16.911 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:16.908+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:17.030 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:17.027+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:17.108 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:17.103+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:17.399 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:17.395+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:17.583 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:17.580+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:17.584 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:17.581+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:17.928 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:17.925+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:18.010 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:18.007+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:18.131 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:18.128+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:18.362 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:18.358+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:18.607 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:18.603+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:18.609 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:18.607+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:18.909 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:18.905+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:19.011 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:19.008+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:19.140 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:19.133+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:19.383 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:19.373+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:19.634 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:19.630+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:19.634 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:19.631+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:19.896 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:19.893+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:20.058 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:20.054+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:20.169 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:20.165+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:20.407 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:20.405+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:20.638 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:20.634+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:20.641 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:20.638+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:20.914 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:28:20.911+0000 1694f700 -1 osd.1 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:05.711607+0000 front 2019-12-12T00:27:05.478363+0000 (oldest deadline 2019-12-12T00:27:47.160340+0000) 2019-12-12T00:28:21.033 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:21.029+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:21.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:28:21.147+0000 1694f700 -1 osd.4 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.199173+0000 front 2019-12-12T00:27:04.130316+0000 (oldest deadline 2019-12-12T00:27:49.414376+0000) 2019-12-12T00:28:21.389 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:28:21.384+0000 1694f700 -1 osd.6 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.383422+0000 front 2019-12-12T00:27:04.539750+0000 (oldest deadline 2019-12-12T00:27:48.314981+0000) 2019-12-12T00:28:21.599 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:28:21.534+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 47sec e713 v713) 2019-12-12T00:28:21.639 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:21.633+0000 1694f700 -1 osd.2 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:03.820281+0000 front 2019-12-12T00:27:03.819260+0000 (oldest deadline 2019-12-12T00:27:48.363492+0000) 2019-12-12T00:28:21.640 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:21.637+0000 1694f700 -1 osd.2 713 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:28:22.053 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:28:22.050+0000 1694f700 -1 osd.7 713 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:27:04.182267+0000 front 2019-12-12T00:27:04.379661+0000 (oldest deadline 2019-12-12T00:27:49.976463+0000) 2019-12-12T00:28:22.642 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:28:22.640+0000 1694f700 -1 osd.2 714 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4518.0:124741 6.f 6.af92fdcf (undecoded) ondisk+read+known_if_redirected e713) 2019-12-12T00:29:05.022 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:29:04.992+0000 114ae700 -1 osd.1 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:05.106 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:29:05.050+0000 3578e700 -1 osd.1 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:05.334 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:29:05.068+0000 3578e700 -1 osd.1 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:05.455 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:29:05.427+0000 31f87700 -1 osd.2 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:05.561 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:29:05.311+0000 32788700 -1 osd.1 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:05.681 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:29:05.543+0000 36790700 -1 osd.2 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:05.720 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:29:05.582+0000 36f91700 -1 osd.2 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:05.735 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:29:05.596+0000 32788700 -1 osd.2 717 build_incremental_map_msg missing incremental map 717 2019-12-12T00:29:41.944 INFO:teuthology.orchestra.run.smithi079.stderr:test_slo_etag_is_hash_of_etags (test.functional.tests.TestSloUTF8) ... ok 2019-12-12T00:29:45.860 INFO:teuthology.orchestra.run.smithi079.stderr:test_slo_get_simple_manifest (test.functional.tests.TestSloUTF8) ... ok 2019-12-12T00:29:49.740 INFO:teuthology.orchestra.run.smithi079.stderr:test_slo_ranged_get (test.functional.tests.TestSloUTF8) ... ok 2019-12-12T00:29:49.740 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-12T00:29:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:---------------------------------------------------------------------- 2019-12-12T00:29:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:Ran 147 tests in 13568.734s 2019-12-12T00:29:49.741 INFO:teuthology.orchestra.run.smithi079.stderr: 2019-12-12T00:29:49.741 INFO:teuthology.orchestra.run.smithi079.stderr:OK 2019-12-12T00:29:49.763 INFO:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph user rm --uid foo.client.0 --purge-data 2019-12-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:29:49.910+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:29:49.933+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:29:49.933+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:29:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:29:49.953+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-12T00:29:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:29:49.953+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:29:49.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:29:49.959+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-12T00:29:49.961 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:29:49.959+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:30:07.184 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:07.181+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:08.070 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:08.066+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:08.183 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:08.180+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:09.055 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:09.050+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:09.177 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:09.173+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:09.219 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:09.215+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:10.081 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:10.077+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:10.155 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:10.151+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:10.244 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:10.239+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:11.084 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:11.076+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:11.180 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:11.177+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:11.266 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:11.262+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:11.335 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:11.332+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:12.087 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:12.083+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:12.141 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:12.138+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:12.255 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:12.249+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:12.345 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:12.338+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:13.089 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:13.085+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:13.098 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:13.096+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:13.235 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:13.231+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:13.386 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:13.383+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:14.047 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:14.043+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:14.064 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:14.060+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:14.229 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:14.224+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:14.403 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:14.400+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:15.007 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:15.003+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:15.090 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:15.087+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:15.201 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:15.196+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:15.431 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:15.419+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:15.993 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:15.987+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:16.086 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:16.083+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:16.231 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:16.226+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:16.380 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:16.376+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:16.975 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:16.971+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:17.132 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:17.129+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:17.250 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:17.246+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:17.426 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:17.423+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:17.952 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:17.948+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:18.151 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:18.148+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:18.282 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:18.279+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:18.470 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:18.466+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:18.970 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:18.966+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:19.202 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:19.199+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:19.309 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:19.306+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:19.439 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:19.436+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:19.973 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:19.969+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:20.231 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:20.227+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:20.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:20.306+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:20.313 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:20.310+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:20.478 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:20.474+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:20.999 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:20.995+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:21.262 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:21.258+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:21.268 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:21.264+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:21.269 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:21.266+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:21.471 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:21.468+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:21.993 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:21.990+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:22.088 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:58:09.020 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.127 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:58:09.058 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.134 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:58:09.066 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.138 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:58:09.070 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.176 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:58:09.081 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.184 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:58:09.088 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.255 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:22.252+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:22.255 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:22.253+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:22.303 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:22.299+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:22.363 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:58:09.288 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.467 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:58:09.399 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.496 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:22.494+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:22.535 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:58:09.439 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:22.966 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:22.963+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:23.221 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:23.217+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:23.223 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:23.221+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:23.345 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:23.342+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:23.465 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:23.463+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:23.962 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:23.959+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:24.208 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:24.205+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:24.209 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:24.207+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:24.354 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:24.351+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:24.441 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:24.438+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:24.929 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:24.925+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:25.212 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:25.208+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:25.213 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:25.210+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:25.313 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:25.310+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:25.412 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:25.407+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:25.913 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:25.909+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:26.253 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:26.250+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:26.254 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:26.251+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:26.271 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:26.267+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:26.387 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:26.383+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:26.868 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:26.864+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:27.281 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:27.275+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:27.281 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:27.277+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:27.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:27.320+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:27.348 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:27.345+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:27.916 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:27.912+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:28.266 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:28.262+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:28.266 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:28.263+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:28.319 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:28.316+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:28.323 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:28.321+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:28.659 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:28.657+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:28.884 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:28.879+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:29.274 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:29.270+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:29.274 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:29.271+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:29.289 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:29.288+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:29.296 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:29.292+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:29.704 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:29.702+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:29.920 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:29.917+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:30.233 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:30.228+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:30.233 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:30.229+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:30.255 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:30.253+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:30.341 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:30.337+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:30.723 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:30.720+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:30.927 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:30.923+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:31.194 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:31.191+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:31.195 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:31.193+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:31.253 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:31.250+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:31.355 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:31.353+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:31.722 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:31.719+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:31.963 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:31.960+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:32.161 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:32.158+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:32.161 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:32.159+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:32.299 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:32.296+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:32.342 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:32.340+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:32.682 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:32.680+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:32.924 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:32.920+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:33.145 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:33.140+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:33.146 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:33.144+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:33.303 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:33.301+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:33.335 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:33.333+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:33.696 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:33.693+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:33.955 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:33.950+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:34.159 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:34.154+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:34.159 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:34.156+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:34.306 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:34.304+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:34.376 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:34.374+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:34.676 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:34.674+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:34.931 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:34.927+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:35.156 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:35.152+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:35.157 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:35.153+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:35.310 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:35.308+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:35.420 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:35.417+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:35.648 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:35.644+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:35.963 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:58:22.895 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:35.971 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:58:22.903 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:35.975 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:58:22.907 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:35.982 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:58:22.913 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:35.982 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:35.978+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:35.990 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:58:22.895 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:36.021 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:58:22.926 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:36.023 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:07:58:22.956 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:36.024 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:07:58:22.956 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:36.131 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:36.128+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:36.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:36.129+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:36.151 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:07:58:23.056 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:30:36.310 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:36.302+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:36.405 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:36.402+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:36.695 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:36.692+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:36.956 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:36.953+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:37.103 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:37.100+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:37.104 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:37.102+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:37.334 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:37.331+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:37.420 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:37.417+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:37.662 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:37.658+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:37.920 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:37.916+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:38.091 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:38.087+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:38.099 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:38.096+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:38.381 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:38.377+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:38.408 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:38.405+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:38.637 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:38.634+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:38.965 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:38.962+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:39.133 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:39.128+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:39.133 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:39.130+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:39.385 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:39.383+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:39.434 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:39.430+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:39.602 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:39.598+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:40.011 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:40.008+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:40.159 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:40.156+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:40.162 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:40.160+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:40.374 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:40.371+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:40.389 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:40.385+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:40.610 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:40.607+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:41.023 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:41.020+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:41.196 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:41.192+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:41.196 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:41.193+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:41.376 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:41.372+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:41.401 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:41.398+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:41.604 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:30:41.601+0000 1694f700 -1 osd.6 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:22.987743+0000 front 2019-12-12T00:29:23.079319+0000 (oldest deadline 2019-12-12T00:30:28.613784+0000) 2019-12-12T00:30:42.000 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:30:41.994+0000 1694f700 -1 osd.2 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:24.453135+0000 front 2019-12-12T00:29:24.655045+0000 (oldest deadline 2019-12-12T00:30:07.183419+0000) 2019-12-12T00:30:42.186 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:42.167+0000 1694f700 -1 osd.1 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:21.767914+0000 front 2019-12-12T00:29:21.428101+0000 (oldest deadline 2019-12-12T00:30:06.671762+0000) 2019-12-12T00:30:42.186 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:42.169+0000 1694f700 -1 osd.1 719 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:42.206 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:30:42.203+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 44sec e719 v719) 2019-12-12T00:30:42.352 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:30:42.349+0000 1694f700 -1 osd.4 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:25.900950+0000 front 2019-12-12T00:29:25.829555+0000 (oldest deadline 2019-12-12T00:30:10.459417+0000) 2019-12-12T00:30:42.361 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:30:42.358+0000 1694f700 -1 osd.7 719 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:29:26.075315+0000 front 2019-12-12T00:29:26.076641+0000 (oldest deadline 2019-12-12T00:30:08.944516+0000) 2019-12-12T00:30:43.291 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:30:43.282+0000 1694f700 -1 osd.1 720 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:4 4.f 4.1eaca32f (undecoded) ondisk+read+known_if_redirected e719) 2019-12-12T00:30:44.271 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:30:44.269+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-12T00:31:03.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.601+0000 7fd9bdcd0ec0 20 RGWRados::pool_iterate: got zonegroup_info.bd8d0a8d-2fda-4de2-80a9-f8429b82bd66 2019-12-12T00:31:03.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.602+0000 7fd9bdcd0ec0 20 RGWRados::pool_iterate: got zone_info.14af9c21-7f56-4de3-9a52-895b5fee19a0 2019-12-12T00:31:03.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.602+0000 7fd9bdcd0ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-12T00:31:03.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.602+0000 7fd9bdcd0ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-12T00:31:03.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.602+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.616 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.614+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-12T00:31:03.617 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.615+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.632+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-12T00:31:03.634 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.632+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.654 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.651+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-12T00:31:03.654 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.651+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.670 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.667+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-12T00:31:03.671 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.668+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.680 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.678+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-12T00:31:03.681 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.678+0000 7fd9bdcd0ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-12T00:31:03.681 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.678+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.688 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.687+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-12T00:31:03.689 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.687+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.709 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.705+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-12T00:31:03.709 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.705+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.724+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-12T00:31:03.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.724+0000 7fd9bdcd0ec0 20 zonegroup default 2019-12-12T00:31:03.726 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.724+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.744+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-12T00:31:03.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.744+0000 7fd9bdcd0ec0 10 Cannot find current period zone using local zone 2019-12-12T00:31:03.746 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.744+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.752+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-12T00:31:03.754 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.752+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.763 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.761+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-12T00:31:03.764 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.761+0000 7fd9bdcd0ec0 20 rados->read ofs=0 len=0 2019-12-12T00:31:03.780 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.777+0000 7fd9bdcd0ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-12T00:31:03.780 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.777+0000 7fd9bdcd0ec0 20 zone default found 2019-12-12T00:31:03.781 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.777+0000 7fd9bdcd0ec0 20 started sync module instance, tier type = 2019-12-12T00:31:03.781 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:03.778+0000 7fd9bdcd0ec0 20 started zone id=14af9c21-7f56-4de3-9a52-895b5fee19a0 (name=default) with tier type = 2019-12-12T00:31:04.126 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.124+0000 7fd9bdcd0ec0 20 add_watcher() i=0 2019-12-12T00:31:04.127 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.124+0000 7fd9bdcd0ec0 20 add_watcher() i=1 2019-12-12T00:31:04.127 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.124+0000 7fd9bdcd0ec0 20 add_watcher() i=2 2019-12-12T00:31:04.127 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.124+0000 7fd9bdcd0ec0 20 add_watcher() i=3 2019-12-12T00:31:04.127 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.124+0000 7fd9bdcd0ec0 20 add_watcher() i=4 2019-12-12T00:31:04.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.124+0000 7fd9bdcd0ec0 20 add_watcher() i=5 2019-12-12T00:31:04.128 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.124+0000 7fd9bdcd0ec0 20 add_watcher() i=6 2019-12-12T00:31:04.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.156+0000 7fd9bdcd0ec0 20 add_watcher() i=7 2019-12-12T00:31:04.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.156+0000 7fd9bdcd0ec0 2 all 8 watchers are set, enabling cache 2019-12-12T00:31:04.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:31:04.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.160+0000 7fd9aaefa700 0 -- 172.21.15.79:0/2693205426 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x55c1c3fae2e0 msgr2=0x55c1c3fb0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2019-12-12T00:31:04.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:04.201+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/250202870 >> [v2:172.21.15.79:6800/9284,v1:172.21.15.79:6801/9284] conn(0x7fd8e4022480 msgr2=0x7fd8e4024990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T00:31:25.731 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:25.536+0000 31f87700 -1 osd.3 724 build_incremental_map_msg missing incremental map 724 2019-12-12T00:31:25.984 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:25.953+0000 31f87700 -1 osd.3 724 build_incremental_map_msg missing incremental map 724 2019-12-12T00:31:26.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:26.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:31:27.449 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:27.388+0000 38f95700 -1 osd.3 725 build_incremental_map_msg missing incremental map 724 2019-12-12T00:31:27.523 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:27.388+0000 38f95700 -1 osd.3 725 build_incremental_map_msg missing incremental map 725 2019-12-12T00:31:28.718 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:28.596+0000 31786700 -1 osd.3 725 build_incremental_map_msg missing incremental map 724 2019-12-12T00:31:28.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:28.596+0000 31786700 -1 osd.3 725 build_incremental_map_msg missing incremental map 725 2019-12-12T00:31:28.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:28.598+0000 31786700 -1 osd.3 725 build_incremental_map_msg missing incremental map 724 2019-12-12T00:31:28.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:31:28.599+0000 31786700 -1 osd.3 725 build_incremental_map_msg missing incremental map 725 2019-12-12T00:31:48.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:31:48.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:32:10.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:32:10.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:32:23.507 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:23.505+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:24.326 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:24.323+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:24.495 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:24.493+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:24.507 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:24.502+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:25.354 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:25.351+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:25.462 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:25.457+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:25.538 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:25.530+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:26.334 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:26.332+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:26.508 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:26.503+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:26.509 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:26.507+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:26.837 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:26.831+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:27.312 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:27.307+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:27.519 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:27.515+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:27.542 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:27.530+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:27.852 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:27.847+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:28.273 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:28.271+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:28.544 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:28.541+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:28.575 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:28.571+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:28.905 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:28.901+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:29.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:29.314+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:29.529 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:29.526+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:29.616 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:29.613+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:29.925 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:29.921+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:30.339 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:30.335+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:30.508 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:30.504+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:30.629 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:30.625+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:30.882 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:30.878+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:31.360 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:31.357+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:31.484 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:31.481+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:31.619 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:31.615+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:31.922 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:31.919+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:32.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:32:32.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:32:32.400 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:32.397+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:32.446 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:32.442+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:32.617 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:32.613+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:32.894 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:32.891+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:33.430 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:33.426+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:33.491 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:33.487+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:33.610 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:33.603+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:33.875 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:33.871+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:34.398 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:34.395+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:34.510 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:34.506+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:34.595 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:34.591+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:34.921 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:34.917+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:35.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:35.379+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:35.478 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:35.474+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:35.596 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:35.591+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:35.962 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:35.959+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:36.414 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:36.410+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:36.444 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:36.441+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:36.566 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:36.562+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:36.943 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:36.940+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:37.423 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:37.420+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:37.463 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:32:37.460+0000 1694f700 -1 osd.6 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.187728+0000 front 2019-12-12T00:31:40.186550+0000 (oldest deadline 2019-12-12T00:32:24.266443+0000) 2019-12-12T00:32:37.521 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:37.517+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:37.976 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:37.973+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:38.378 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:38.374+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:38.553 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:38.549+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:38.935 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:38.932+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:39.428 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:39.425+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:39.551 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:39.547+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:39.941 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:39.938+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:40.440 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:40.436+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:40.590 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:40.586+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:40.901 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:40.897+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:41.470 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:41.466+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:41.590 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:41.586+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:41.929 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:41.926+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:42.452 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:42.449+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:42.617 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:42.611+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:42.951 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:42.946+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:43.499 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:43.495+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:43.666 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:43.661+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:43.975 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:43.971+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:44.464 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:44.461+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:44.708 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:44.703+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:44.975 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:44.971+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:45.512 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:45.508+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:45.692 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:45.688+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:46.003 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:46.000+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:46.477 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:46.471+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:46.646 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:46.642+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:47.023 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:47.020+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:47.451 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:47.449+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:47.620 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:47.617+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:48.041 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:48.038+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:48.451 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:48.447+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:48.629 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:48.620+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:49.047 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:49.044+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:49.406 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:49.403+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:49.608 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:49.604+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:50.033 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:50.030+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:50.384 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:50.381+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:50.599 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:50.595+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:51.004 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:51.001+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:51.379 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:51.376+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:51.599 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:51.595+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:52.021 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:52.017+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:52.345 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:52.341+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:52.567 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:52.564+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:52.989 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:52.986+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:53.305 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:53.302+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:53.537 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:53.532+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:54.022 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:54.018+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:54.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:32:54.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:32:54.304 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:54.301+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:54.503 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:54.499+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:55.004 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:55.001+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:55.287 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:55.283+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:55.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:55.454+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:55.981 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:55.978+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:56.317 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:56.314+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:56.500 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:56.497+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:56.994 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:56.982+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:57.364 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:57.360+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:57.548 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:57.544+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:57.659 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:32:57.656+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 44sec e725 v725) 2019-12-12T00:32:58.000 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:57.996+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:58.365 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:58.361+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:58.600 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:58.595+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:32:59.044 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:32:59.040+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:32:59.366 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:32:59.362+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:32:59.635 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:32:59.632+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:33:00.085 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:33:00.081+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:33:00.405 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:33:00.402+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:33:00.652 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:33:00.645+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:33:00.864 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:33:00.861+0000 1694f700 -1 osd.7 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.372358+0000 front 2019-12-12T00:31:40.379124+0000 (oldest deadline 2019-12-12T00:33:00.000762+0000) 2019-12-12T00:33:01.112 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:33:01.107+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:33:01.456 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:33:01.453+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:33:01.635 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:33:01.631+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:33:01.879 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:33:01.875+0000 1694f700 -1 osd.7 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.372358+0000 front 2019-12-12T00:31:40.379124+0000 (oldest deadline 2019-12-12T00:33:00.000762+0000) 2019-12-12T00:33:02.076 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:33:02.073+0000 1694f700 -1 osd.2 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.371138+0000 front 2019-12-12T00:31:40.220156+0000 (oldest deadline 2019-12-12T00:32:26.087395+0000) 2019-12-12T00:33:02.501 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:33:02.497+0000 1694f700 -1 osd.4 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:41.638385+0000 front 2019-12-12T00:31:41.653337+0000 (oldest deadline 2019-12-12T00:32:23.227872+0000) 2019-12-12T00:33:02.594 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:33:02.590+0000 1694f700 -1 osd.1 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:42.322150+0000 front 2019-12-12T00:31:42.190440+0000 (oldest deadline 2019-12-12T00:32:24.391135+0000) 2019-12-12T00:33:02.722 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:33:02.719+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 44sec e725 v725) 2019-12-12T00:33:02.865 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:33:02.863+0000 1694f700 -1 osd.7 725 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:31:40.372358+0000 front 2019-12-12T00:31:40.379124+0000 (oldest deadline 2019-12-12T00:33:00.000762+0000) 2019-12-12T00:33:16.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:33:16.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:33:38.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:33:38.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:33:38.683 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:33:38.673+0000 38f95700 -1 osd.3 727 build_incremental_map_msg missing incremental map 724 2019-12-12T00:33:38.684 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:33:38.674+0000 38f95700 -1 osd.3 727 build_incremental_map_msg missing incremental map 725 2019-12-12T00:33:39.560 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:33:39.509+0000 31f87700 -1 osd.2 729 build_incremental_map_msg missing incremental map 729 2019-12-12T00:33:39.801 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:33:39.615+0000 36f91700 -1 osd.2 729 build_incremental_map_msg missing incremental map 729 2019-12-12T00:33:39.854 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:33:39.654+0000 32788700 -1 osd.2 729 build_incremental_map_msg missing incremental map 729 2019-12-12T00:33:40.105 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:33:39.874+0000 36790700 -1 osd.2 729 build_incremental_map_msg missing incremental map 729 2019-12-12T00:34:00.159 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:34:00.157+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:34:22.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:34:22.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:34:37.216 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:37.212+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:38.190 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:38.186+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:38.248 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:38.243+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:39.231 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:39.226+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:39.233 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:39.229+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:39.618 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:39.608+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:40.189 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:40.186+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:40.231 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:40.227+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:40.617 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:40.614+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:41.228 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:41.225+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:41.239 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:41.235+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:41.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:41.594+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:42.220 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:42.217+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:42.237 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:42.233+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:42.600 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:42.596+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:42.741 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:42.738+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:43.217 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:43.214+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:43.258 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:43.255+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:43.577 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:43.573+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:43.781 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:43.776+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:44.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:34:44.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:34:44.260 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:44.256+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:44.303 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:44.300+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:44.591 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:44.588+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:44.749 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:44.744+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:45.222 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:45.218+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:45.348 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:45.344+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:45.597 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:45.594+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:45.728 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:45.726+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:46.205 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:46.202+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:46.349 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:46.346+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:46.588 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:46.584+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:46.719 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:46.714+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:47.218 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:47.214+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:47.311 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:47.307+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:47.578 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:47.575+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:47.673 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:47.669+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:48.188 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:48.184+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:48.329 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:48.326+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:48.631 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:48.619+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:48.666 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:48.656+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:49.161 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:02:36.093 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.173 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:02:36.100 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.176 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:02:36.108 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.183 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:02:36.115 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.186 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:02:36.090 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.190 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:02:36.094 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.197 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:49.193+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:49.375 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:49.372+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:49.499 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:02:36.429 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.539 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:02:36.444 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.577 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:02:36.509 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:34:49.621 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:49.619+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:49.698 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:49.695+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:50.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:50.207+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:50.350 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:50.347+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:50.665 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:50.663+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:50.741 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:50.738+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:51.185 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:51.176+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:51.339 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:51.335+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:51.670 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:51.666+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:51.736 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:51.732+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:52.169 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:52.166+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:52.334 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:52.329+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:52.641 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:52.637+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:52.761 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:52.758+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:53.179 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:53.176+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:53.380 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:53.376+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:53.647 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:53.637+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:53.809 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:53.806+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:54.195 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:54.191+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:54.408 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:54.405+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:54.693 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:54.689+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:54.855 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:54.851+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:55.223 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:55.220+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:55.417 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:55.414+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:55.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:55.724+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:55.854 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:55.851+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:56.240 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:56.236+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:56.399 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:56.394+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:56.757 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:56.753+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:56.830 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:56.826+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:57.271 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:57.268+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:57.395 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:57.390+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:57.743 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:57.739+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:57.813 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:57.807+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:58.227 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:58.224+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:58.395 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:58.391+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:58.759 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:58.755+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:58.784 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:58.781+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:34:59.243 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:34:59.230+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:34:59.426 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:34:59.415+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:34:59.728 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:34:59.725+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:34:59.803 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:34:59.800+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:00.200 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:00.196+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:00.473 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:00.469+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:00.778 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:00.775+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:00.833 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:00.829+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:01.223 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:01.220+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:01.451 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:01.447+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:01.748 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:01.745+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:01.798 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:01.796+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:02.271 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:02.268+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:02.464 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:02.461+0000 1694f700 -1 osd.1 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:02.720 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:02.716+0000 1694f700 -1 osd.7 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:02.850 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:02.846+0000 1694f700 -1 osd.6 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:03.239 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:03.229+0000 1694f700 -1 osd.4 731 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:03.686 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:03.498+0000 1694f700 -1 osd.1 732 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:03.729 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:03.695+0000 1694f700 -1 osd.7 732 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:04.120 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:04.015+0000 1694f700 -1 osd.6 732 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:04.212 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:04.209+0000 1694f700 -1 osd.4 732 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:04.752 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:04.630+0000 1694f700 -1 osd.1 733 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:04.884 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:04.875+0000 1694f700 -1 osd.7 733 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:05.137 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:05.135+0000 1694f700 -1 osd.6 733 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:05.252 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:05.250+0000 1694f700 -1 osd.4 733 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:05.995 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:05.929+0000 1694f700 -1 osd.7 734 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:06.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:35:06.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:35:06.175 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:06.172+0000 1694f700 -1 osd.6 734 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:06.230 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:06.226+0000 1694f700 -1 osd.4 734 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:06.282 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:06.155+0000 1694f700 -1 osd.1 734 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:07.193 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:07.188+0000 1694f700 -1 osd.7 735 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:07.202 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:07.195+0000 1694f700 -1 osd.6 735 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:07.223 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:07.204+0000 1694f700 -1 osd.1 735 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:07.624 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:07.538+0000 1694f700 -1 osd.4 735 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:08.126 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:08.118+0000 1694f700 -1 osd.2 736 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:08.282 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:08.208+0000 1694f700 -1 osd.6 736 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:08.325 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:08.321+0000 1694f700 -1 osd.1 736 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:08.463 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:08.458+0000 1694f700 -1 osd.7 736 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:08.983 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:08.977+0000 1694f700 -1 osd.4 736 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:09.123 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:09.103+0000 1694f700 -1 osd.2 736 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:09.256 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:09.253+0000 1694f700 -1 osd.6 737 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:09.458 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:09.395+0000 1694f700 -1 osd.1 737 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:09.514 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:09.510+0000 1694f700 -1 osd.7 737 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:10.037 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:10.004+0000 1694f700 -1 osd.4 737 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:10.306 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:10.304+0000 1694f700 -1 osd.6 738 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:10.376 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:10.290+0000 1694f700 -1 osd.2 738 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:10.531 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:10.522+0000 1694f700 -1 osd.1 737 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:10.546 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:10.540+0000 1694f700 -1 osd.7 738 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:11.133 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:11.104+0000 1694f700 -1 osd.4 739 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:11.284 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:11.281+0000 1694f700 -1 osd.6 739 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:11.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:11.350+0000 1694f700 -1 osd.2 739 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:11.538 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:11.535+0000 1694f700 -1 osd.7 739 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:11.773 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:11.711+0000 1694f700 -1 osd.1 739 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:12.153 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:12.150+0000 1694f700 -1 osd.4 740 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:12.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:12.234+0000 1694f700 -1 osd.6 740 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:12.340 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:12.336+0000 1694f700 -1 osd.2 740 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:12.520 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:12.503+0000 1694f700 -1 osd.7 740 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:12.662 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:12.652+0000 31786700 -1 osd.2 740 build_incremental_map_msg missing incremental map 717 2019-12-12T00:35:12.664 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:12.652+0000 31786700 -1 osd.2 740 build_incremental_map_msg missing incremental map 729 2019-12-12T00:35:12.698 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:12.695+0000 1694f700 -1 osd.1 740 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:13.105 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:13.103+0000 1694f700 -1 osd.4 740 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:13.235 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:35:13.231+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e731 v731) 2019-12-12T00:35:13.266 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:13.264+0000 1694f700 -1 osd.6 740 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:13.327 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:13.314+0000 1694f700 -1 osd.2 741 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:13.519 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:13.516+0000 1694f700 -1 osd.7 742 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:13.679 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:13.676+0000 1694f700 -1 osd.1 741 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:14.246 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:14.244+0000 1694f700 -1 osd.6 742 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:14.261 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:14.221+0000 1694f700 -1 osd.4 742 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:14.304 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:14.300+0000 1694f700 -1 osd.2 742 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:14.572 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:14.486+0000 1694f700 -1 osd.7 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:14.718 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:14.692+0000 1694f700 -1 osd.1 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:15.239 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:15.238+0000 1694f700 -1 osd.6 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:15.305 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:15.286+0000 1694f700 -1 osd.2 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:15.382 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:15.371+0000 1694f700 -1 osd.4 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:15.685 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:15.682+0000 1694f700 -1 osd.7 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:15.757 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:15.732+0000 1694f700 -1 osd.1 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:16.195 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:16.193+0000 1694f700 -1 osd.6 743 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:16.395 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:16.327+0000 1694f700 -1 osd.2 744 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:16.684 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:16.586+0000 1694f700 -1 osd.4 744 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:16.697 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:16.694+0000 1694f700 -1 osd.1 744 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:16.828 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:16.733+0000 1694f700 -1 osd.7 744 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:17.207 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:17.205+0000 1694f700 -1 osd.6 744 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:17.404 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:17.375+0000 1694f700 -1 osd.2 745 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:17.685 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:17.600+0000 1694f700 -1 osd.4 745 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:17.768 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:17.735+0000 1694f700 -1 osd.1 745 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.016575+0000 front 2019-12-12T00:33:57.160191+0000 (oldest deadline 2019-12-12T00:34:38.062562+0000) 2019-12-12T00:35:17.923 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:17.921+0000 1694f700 -1 osd.7 745 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:18.202 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:18.198+0000 1694f700 -1 osd.6 745 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:59.482702+0000 front 2019-12-12T00:33:59.407590+0000 (oldest deadline 2019-12-12T00:34:41.935714+0000) 2019-12-12T00:35:18.331 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:35:18.327+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 42sec e731 v731) 2019-12-12T00:35:18.413 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:18.407+0000 1694f700 -1 osd.2 745 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:18.907 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:18.903+0000 1694f700 -1 osd.7 746 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:53.174989+0000 front 2019-12-12T00:33:53.127959+0000 (oldest deadline 2019-12-12T00:34:38.995248+0000) 2019-12-12T00:35:18.921 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:35:18.802+0000 1694f700 -1 osd.4 746 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.589381+0000 front 2019-12-12T00:33:59.707452+0000 (oldest deadline 2019-12-12T00:34:36.425485+0000) 2019-12-12T00:35:19.397 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:35:19.367+0000 1694f700 -1 osd.2 746 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:33:57.083003+0000 front 2019-12-12T00:33:57.224347+0000 (oldest deadline 2019-12-12T00:35:08.014063+0000) 2019-12-12T00:35:28.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:35:28.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:35:28.595 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:03:15.527 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.613 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:03:15.545 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.650 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:03:15.555 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.683 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:03:15.614 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.685 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:03:15.590 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.692 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:03:15.623 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.766 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:03:15.698 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.837 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:03:15.769 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:28.853 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:03:15.758 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:35:33.619 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:33.617+0000 1694f700 -1 osd.6 755 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:34.493 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:35:34.487+0000 37f93700 -1 osd.1 755 build_incremental_map_msg missing incremental map 717 2019-12-12T00:35:34.628 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:34.625+0000 1694f700 -1 osd.6 755 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:35.660 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:35.655+0000 1694f700 -1 osd.6 756 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:36.708 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:36.637+0000 1694f700 -1 osd.6 757 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:37.596 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:37.594+0000 1694f700 -1 osd.6 757 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:38.790 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:38.760+0000 1694f700 -1 osd.6 758 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:39.768 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:39.766+0000 1694f700 -1 osd.6 759 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:40.956 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:40.854+0000 1694f700 -1 osd.6 760 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:42.156 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:42.097+0000 1694f700 -1 osd.6 761 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:43.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:43.072+0000 1694f700 -1 osd.6 762 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:44.055 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:44.053+0000 1694f700 -1 osd.6 762 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:45.074 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:45.027+0000 1694f700 -1 osd.6 763 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:46.396 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:35:46.327+0000 1694f700 -1 osd.6 764 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.5088.0:438 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94290704980672 gen 1] snapc 0=[] ondisk+write+known_if_redirected e732) 2019-12-12T00:35:50.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:35:50.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:35:50.801 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:50.798+0000 1694f700 -1 osd.7 767 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:51.792 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:51.788+0000 1694f700 -1 osd.7 767 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:52.935 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:52.847+0000 1694f700 -1 osd.7 768 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:53.966 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:53.960+0000 1694f700 -1 osd.7 769 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:54.031 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-12T00:35:54.006+0000 20e25700 -1 RGWWatcher::handle_error cookie 821300688 err (110) Connection timed out 2019-12-12T00:35:54.961 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:54.958+0000 1694f700 -1 osd.7 770 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:55.957 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:55.955+0000 1694f700 -1 osd.7 770 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:57.000 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:56.996+0000 1694f700 -1 osd.7 771 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:58.062 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:58.039+0000 1694f700 -1 osd.7 771 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:35:59.387 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:35:59.263+0000 1694f700 -1 osd.7 772 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:00.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:00.270+0000 1694f700 -1 osd.7 772 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:01.283 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:01.280+0000 1694f700 -1 osd.7 773 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:02.340 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:02.325+0000 1694f700 -1 osd.7 773 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:03.368 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:03.364+0000 1694f700 -1 osd.7 774 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:04.346 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:04.343+0000 1694f700 -1 osd.7 775 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:05.367 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:05.365+0000 1694f700 -1 osd.7 776 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:06.371 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:06.367+0000 1694f700 -1 osd.7 776 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:07.416 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:07.413+0000 1694f700 -1 osd.7 776 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:08.389 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:08.386+0000 1694f700 -1 osd.7 776 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:09.394 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:09.388+0000 1694f700 -1 osd.7 776 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:10.360 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:10.357+0000 1694f700 -1 osd.7 776 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:10.872 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:36:10.786+0000 1694f700 -1 osd.3 767 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4518.0:125016 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e729) 2019-12-12T00:36:11.416 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:11.414+0000 1694f700 -1 osd.7 776 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:12.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:36:12.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:36:12.849 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:12.643+0000 1694f700 -1 osd.7 777 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:13.888 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:13.822+0000 1694f700 -1 osd.7 778 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:14.874 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:14.872+0000 1694f700 -1 osd.7 779 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:15.874 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:15.871+0000 1694f700 -1 osd.7 780 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:16.837 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:16.835+0000 1694f700 -1 osd.7 781 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:17.888 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:17.884+0000 1694f700 -1 osd.7 782 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:18.929 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:18.927+0000 1694f700 -1 osd.7 783 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:19.955 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:19.951+0000 1694f700 -1 osd.7 784 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:21.280 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:21.233+0000 1694f700 -1 osd.7 785 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:22.306 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:22.250+0000 1694f700 -1 osd.7 786 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:23.299 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:23.297+0000 1694f700 -1 osd.7 787 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:24.418 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:24.410+0000 1694f700 -1 osd.7 787 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:25.399 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:25.397+0000 1694f700 -1 osd.7 788 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:26.428 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:26.424+0000 1694f700 -1 osd.7 788 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:27.427 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:27.425+0000 1694f700 -1 osd.7 789 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:28.432 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:28.429+0000 1694f700 -1 osd.7 790 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:29.904 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:36:29.845+0000 1694f700 -1 osd.7 790 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4518.0:125002 7.f 7.c4474f6f (undecoded) ondisk+retry+write+known_if_redirected e747) 2019-12-12T00:36:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:36:34.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:36:56.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:36:56.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:37:05.173 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:05.170+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:06.134 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:06.132+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:07.146 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:07.144+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:08.105 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:08.103+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:09.117 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:09.116+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:10.095 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:10.094+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:11.120 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:11.118+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:12.142 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:12.140+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:13.111 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:13.109+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:14.161 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:14.159+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:15.176 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:15.173+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:16.208 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:16.206+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:17.207 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:17.204+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:18.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:37:18.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:37:18.186 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:18.184+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:19.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:19.185+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:20.211 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:20.208+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:21.259 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:21.256+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:22.313 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:22.310+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:23.277 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:23.274+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:24.258 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:24.256+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:25.245 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:25.242+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:26.216 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:26.213+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:27.241 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:27.238+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:27.592 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:27.588+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:28.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:28.247+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:28.251 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:28.248+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:28.585 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:28.580+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:28.963 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:28.960+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:29.222 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:29.219+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:29.232 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:29.224+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:29.615 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:29.609+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:30.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:30.024+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:30.085 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:30.079+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:30.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:30.177+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:30.182 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:30.179+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:30.645 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:30.640+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:31.059 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:31.055+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:31.135 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:31.129+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:31.176 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:31.171+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:31.224 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:31.220+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:31.224 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:31.222+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:31.690 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:31.685+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:32.083 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:32.079+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:32.179 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:32.173+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:32.197 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:32.193+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:32.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:32.259+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:32.265 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:32.261+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:32.692 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:32.687+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:33.126 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:33.122+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:33.208 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:33.203+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:33.231 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:33.227+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:33.232 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:33.229+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:33.241 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:33.237+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:33.626 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:05:20.558 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.634 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:05:20.566 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.648 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:05:20.579 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.652 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:05:20.584 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.662 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:05:20.566 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.675 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:33.670+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:33.690 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:05:20.594 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.881 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:05:20.812 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.916 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:05:20.848 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:33.999 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:05:20.904 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:34.171 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:34.167+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:34.172 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:34.169+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:34.247 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:34.243+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:34.247 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:34.244+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:34.267 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:34.264+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:34.651 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:34.646+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:35.191 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:35.188+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:35.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:35.195+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:35.219 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:35.217+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:35.220 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:35.218+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:35.261 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:35.256+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:35.607 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:35.604+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:36.162 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:36.159+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:36.215 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:36.210+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:36.223 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:36.219+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:36.223 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:36.220+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:36.224 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:36.221+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:36.566 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:36.562+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:37.140 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:37.136+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:37.182 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:37.177+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:37.188 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:37.185+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:37.191 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:37.187+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:37.213 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:37.209+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:37.608 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:37.603+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:38.153 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:38.149+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:38.167 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:38.164+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:38.169 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:38.165+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:38.170 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:38.167+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:38.196 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:38.193+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:38.622 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:38.618+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:39.108 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:39.104+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:39.196 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:39.190+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:39.218 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:39.214+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:39.219 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:39.216+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:39.234 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:39.231+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:39.654 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:39.650+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:40.071 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:40.066+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:40.160 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:37:40.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:37:40.173 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:40.169+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:40.243 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:40.239+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:40.244 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:40.241+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:40.275 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:40.271+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:40.706 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:40.702+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:41.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:41.040+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:41.201 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:41.197+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:41.247 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:41.243+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:41.247 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:41.245+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:41.276 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:41.271+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:41.759 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:41.754+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:42.098 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:42.090+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:42.158 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:42.153+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:42.271 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:42.268+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:42.272 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:42.269+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:42.322 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:42.318+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:42.750 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:42.745+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:43.106 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:43.102+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:43.177 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:43.174+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:43.281 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:43.277+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:43.281 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:43.279+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:43.317 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:43.313+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:43.788 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:43.784+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:44.065 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:44.061+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:44.224 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:44.221+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:44.334 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:44.330+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:44.334 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:44.331+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:44.366 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:44.363+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:44.795 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:44.790+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:45.044 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:45.041+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:45.190 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:45.186+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:45.332 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:45.329+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:45.355 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:45.352+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:45.355 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:45.353+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:45.757 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:45.753+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:46.072 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:46.069+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:46.178 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:46.174+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:46.319 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:46.315+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:46.320 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:46.317+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:46.333 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:46.329+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:46.772 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:46.767+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:47.104 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:47.100+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:47.156 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:47.153+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:47.307 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:47.303+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:47.308 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:47.304+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:47.341 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:47.337+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:47.757 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:47.753+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:48.076 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:48.072+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:48.194 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:48.190+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:48.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:48.310+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:48.314 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:48.312+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:48.331 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:48.327+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:48.793 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:48.788+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:49.045 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:49.040+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:49.165 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:49.160+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:49.306 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:49.302+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:49.307 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:49.304+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:49.356 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:49.352+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:49.746 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:49.743+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:50.049 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:50.043+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:50.138 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:50.131+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:50.304 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:50.299+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:50.304 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:50.301+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:50.392 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:50.389+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:50.779 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:50.774+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:51.015 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:51.011+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:51.183 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:51.179+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:51.356 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:51.352+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:51.357 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:51.354+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:51.405 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:51.400+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:51.786 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:51.782+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:52.060 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:52.056+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:52.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:52.165+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:52.341 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:52.337+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:52.342 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:52.338+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:52.438 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:52.433+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:52.779 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:52.775+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:53.032 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:53.026+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:53.132 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:53.128+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:53.383 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:53.379+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:53.384 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:53.381+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:53.458 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:53.453+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:53.769 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:53.765+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:54.012 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:54.008+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:54.119 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:54.110+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:54.391 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:54.387+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:54.392 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:54.389+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:54.495 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:54.492+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:54.638 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:05:41.570 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.657 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:05:41.588 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.676 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:05:41.608 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.678 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:05:41.610 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.722 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:05:41.654 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.726 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:05:41.658 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.738 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:05:41.643 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.741 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:54.737+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:54.750 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:05:41.654 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.827 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:05:41.732 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:37:54.985 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:54.982+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:55.168 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:55.164+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:55.370 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:55.367+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:55.370 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:55.368+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:55.482 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:55.478+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:55.720 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:55.714+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:55.978 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:55.974+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:56.122 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:56.119+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:56.391 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:56.388+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:56.391 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:56.389+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:56.440 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:56.436+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:56.741 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:56.738+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:57.002 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:56.998+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:57.087 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:57.083+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:57.370 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:57.366+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:57.371 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:57.368+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:57.449 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:57.446+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:57.713 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:57.709+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:57.972 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:57.969+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:58.072 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:58.068+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:58.407 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:58.402+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:58.407 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:58.404+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:58.489 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:58.485+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:58.728 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:58.724+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:58.930 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:58.927+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:37:59.028 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:37:59.024+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:37:59.429 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:59.425+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:37:59.430 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:37:59.427+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:37:59.493 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:37:59.489+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:37:59.585 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:37:59.578+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e790 v790) 2019-12-12T00:37:59.711 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:37:59.707+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:37:59.889 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:37:59.885+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:00.043 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:00.033+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:00.425 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:00.421+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:00.426 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:00.423+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:00.455 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:00.451+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:00.723 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:00.719+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:00.906 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:00.903+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:01.042 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:01.038+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:01.428 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:01.421+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:01.476 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:01.474+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:01.482 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:01.476+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:01.693 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:01.689+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:01.926 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:01.922+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:02.032 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:02.028+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:02.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:38:02.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:38:02.389 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:02.384+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:02.465 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:02.461+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:02.466 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:02.463+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:02.711 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:02.707+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:02.970 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:02.962+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:03.067 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:03.063+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:03.410 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:03.402+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:03.440 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:03.436+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:03.440 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:03.438+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:03.690 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:03.682+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:03.921 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:03.918+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:04.083 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:04.078+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:04.415 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:04.411+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:04.447 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:04.443+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:04.447 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:04.445+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:04.601 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:38:04.594+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e790 v790) 2019-12-12T00:38:04.674 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:04.671+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:04.912 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:04.909+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:05.127 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:05.123+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:05.462 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:05.458+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:05.463 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:05.460+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:05.463 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:05.462+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:05.628 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:05.624+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:05.963 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:05.958+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:06.099 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:06.096+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:06.469 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:06.466+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:06.469 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:06.467+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:06.478 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:06.474+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:06.682 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:06.670+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:06.969 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:06.966+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:07.070 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:07.066+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:07.472 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:07.468+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:07.507 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:07.501+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:07.507 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:07.504+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:07.718 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:07.714+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:08.014 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:08.011+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:08.053 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:08.050+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:08.441 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:08.437+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:08.526 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:08.523+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:08.526 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:08.525+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:08.754 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:08.749+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:09.032 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:38:09.029+0000 1694f700 -1 osd.4 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.342241+0000 front 2019-12-12T00:36:45.520119+0000 (oldest deadline 2019-12-12T00:37:31.103306+0000) 2019-12-12T00:38:09.034 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:38:09.031+0000 1694f700 -1 osd.7 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:46.553198+0000 front 2019-12-12T00:36:46.617710+0000 (oldest deadline 2019-12-12T00:37:28.196744+0000) 2019-12-12T00:38:09.454 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:38:09.449+0000 1694f700 -1 osd.2 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.040652+0000 front 2019-12-12T00:36:45.273201+0000 (oldest deadline 2019-12-12T00:37:29.666398+0000) 2019-12-12T00:38:09.500 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:09.497+0000 1694f700 -1 osd.6 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:42.554605+0000 front 2019-12-12T00:36:42.502785+0000 (oldest deadline 2019-12-12T00:37:27.780746+0000) 2019-12-12T00:38:09.501 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:09.499+0000 1694f700 -1 osd.6 790 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+write+known_if_redirected e790) 2019-12-12T00:38:09.687 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:38:09.682+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 46sec e790 v790) 2019-12-12T00:38:09.760 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:38:09.754+0000 1694f700 -1 osd.1 790 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:36:45.719468+0000 front 2019-12-12T00:36:45.722677+0000 (oldest deadline 2019-12-12T00:37:27.318190+0000) 2019-12-12T00:38:11.228 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:11.218+0000 1694f700 -1 osd.6 791 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:12.305 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:12.273+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7.1d5e1897 (undecoded) ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:13.244 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:13.241+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:14.247 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:14.244+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:15.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:15.261+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:16.285 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:16.283+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:17.331 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:17.328+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:18.143 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:38:18.137+0000 1a89c700 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg recovery_unfound (PG_DAMAGED) 2019-12-12T00:38:18.345 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:18.343+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:19.381 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:19.378+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:20.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:20.379+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:21.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:21.341+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:22.387 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:22.385+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:23.369 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:23.366+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:24.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:38:24.158+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:38:24.383 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:24.377+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:25.385 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:25.382+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:26.419 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:26.416+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:27.450 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:27.448+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:28.500 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:28.498+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:29.539 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:29.537+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:30.538 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:30.535+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:31.497 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:31.494+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:32.472 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:32.470+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:33.483 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:33.480+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:34.457 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:34.455+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:35.413 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:35.411+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:36.374 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:38:36.371+0000 1694f700 -1 osd.6 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:89253 7.7 7:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e790) 2019-12-12T00:38:44.913 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:06:31.845 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:44.935 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:06:31.866 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:44.937 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:06:31.869 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:44.947 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:06:31.878 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:45.005 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:06:31.909 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:45.015 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:06:31.919 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:45.033 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:06:31.965 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:45.035 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:06:31.967 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:45.140 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:06:32.044 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:38:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:38:46.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:38:58.392 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:38:58.239+0000 36f91700 -1 osd.3 795 build_incremental_map_msg missing incremental map 795 2019-12-12T00:39:01.058 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:39:00.882+0000 38f95700 -1 osd.3 797 build_incremental_map_msg missing incremental map 795 2019-12-12T00:39:01.058 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:39:00.885+0000 38f95700 -1 osd.3 797 build_incremental_map_msg missing incremental map 795 2019-12-12T00:39:08.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:39:08.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:39:30.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:39:30.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:39:49.055 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:49.052+0000 1694f700 -1 osd.6 807 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:50.119 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:50.072+0000 1694f700 -1 osd.6 808 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:51.074 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:51.071+0000 1694f700 -1 osd.6 809 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:52.119 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:52.116+0000 1694f700 -1 osd.6 809 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:52.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:39:52.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:39:53.073 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:53.070+0000 1694f700 -1 osd.6 809 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:54.105 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:54.102+0000 1694f700 -1 osd.6 810 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:55.284 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:55.186+0000 1694f700 -1 osd.6 811 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:56.147 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:56.144+0000 1694f700 -1 osd.6 811 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:57.234 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:57.233+0000 1694f700 -1 osd.6 812 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:58.309 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:58.227+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:39:59.266 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:39:59.264+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:00.316 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:00.314+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:01.285 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:01.284+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:02.291 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:02.289+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:03.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:03.295+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:04.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:04.281+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:05.321 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:05.319+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:06.335 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:06.332+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:07.300 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:07.297+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:08.264 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:08.259+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:09.184 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:09.180+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:09.324 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:09.314+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:10.230 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:10.226+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:10.295 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:10.292+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:10.419 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:10.415+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:11.178 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:11.173+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:11.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:11.181+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:11.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:11.340+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:11.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:11.342+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:11.400 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:11.396+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:12.136 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:12.131+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:12.147 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:12.144+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:12.371 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:12.363+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:12.371 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:12.365+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:12.384 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:12.380+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:12.867 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:12.862+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:13.121 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:13.117+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:13.174 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:13.171+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:13.371 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:13.367+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:13.372 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:13.369+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:13.415 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:13.411+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:13.843 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:13.840+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:14.161 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:14.156+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:14.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:40:14.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:40:14.198 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:14.194+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:14.403 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:14.399+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:14.406 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:14.390+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:14.407 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:14.392+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:14.801 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:14.797+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:15.159 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:15.155+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:15.201 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:15.197+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:15.387 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:15.384+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:15.388 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:15.385+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:15.425 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:15.421+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:15.849 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:15.845+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:16.132 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:16.127+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:16.176 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:16.173+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:16.376 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:16.371+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:16.377 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:16.373+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:16.466 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:16.457+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:16.815 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:16.813+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:17.133 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:17.130+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:17.200 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:17.195+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:17.374 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:17.370+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:17.375 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:17.372+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:17.420 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:17.417+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:17.770 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:17.766+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:18.152 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:18.149+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:18.266 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:18.248+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:18.329 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:18.326+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:18.330 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:18.327+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:18.460 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:18.455+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:18.766 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:18.762+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:19.194 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:19.190+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:19.253 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:19.249+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:19.305 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:19.297+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:19.305 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:19.299+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:19.450 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:19.445+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:19.775 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:19.771+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:20.209 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:20.205+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:20.247 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:20.243+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:20.300 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:20.296+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:20.301 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:20.298+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:20.479 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:20.475+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:20.775 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:20.772+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:21.181 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:21.176+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:21.303 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:21.299+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:21.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:21.313+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:21.318 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:21.315+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:21.450 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:21.443+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:21.806 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:21.802+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:22.188 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:22.183+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:22.277 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:22.271+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:22.339 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:22.335+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:22.340 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:22.337+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:22.437 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:22.427+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:22.789 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:22.785+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:23.214 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:23.210+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:23.261 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:23.257+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:23.357 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:23.353+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:23.358 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:23.355+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:23.407 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:23.402+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:23.828 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:23.824+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:24.220 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:24.216+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:24.243 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:24.239+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:24.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:24.398+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:24.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:24.400+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:24.448 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:24.443+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:24.814 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:24.810+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:25.196 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:25.192+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:25.215 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:25.209+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:25.433 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:25.429+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:25.434 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:25.431+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:25.443 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:25.432+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:25.802 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:25.799+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:26.160 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:26.155+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:26.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:26.187+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:26.430 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:26.426+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:26.431 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:26.428+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:26.476 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:26.470+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:26.785 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:26.782+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:27.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:27.174+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:27.188 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:27.182+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:27.443 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:27.438+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:27.444 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:27.440+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:27.514 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:27.509+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:27.750 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:27.745+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:28.145 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:28.141+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:28.146 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:28.142+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:28.469 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:28.463+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:28.470 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:28.466+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:28.470 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:28.468+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:28.780 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:28.777+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:29.157 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:29.153+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:29.195 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:29.190+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:29.441 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:29.436+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:29.441 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:29.438+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:29.509 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:29.504+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:29.784 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:29.780+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:30.178 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:30.167+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:30.179 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:30.172+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:30.427 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:30.423+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:30.427 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:30.424+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:30.516 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:30.510+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:30.754 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:30.750+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:31.200 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:31.196+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:31.210 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:31.206+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:31.400 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:31.396+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:31.401 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:31.398+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:31.552 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:31.547+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:31.746 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:31.741+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:32.212 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:32.208+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:32.256 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:32.252+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:32.376 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:32.371+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:32.376 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:32.373+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:32.582 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:32.578+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:32.744 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:32.741+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:33.178 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:33.175+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:33.221 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:33.216+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:33.362 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:33.357+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:33.363 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:33.360+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:33.571 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:33.566+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:33.712 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:33.708+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:34.179 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:34.175+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:34.228 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:34.223+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:34.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:34.313+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:34.317 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:34.315+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:34.588 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:34.582+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:34.729 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:34.723+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:35.182 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:35.178+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:35.213 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:35.209+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:35.296 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:35.292+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:35.297 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:35.294+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:35.593 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:35.588+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:35.776 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:35.773+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:36.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:40:36.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:40:36.171 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:36.167+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:36.242 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:36.238+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:36.325 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:36.321+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:36.326 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:36.323+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:36.629 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:36.624+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:36.763 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:36.757+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:37.173 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:37.169+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:37.224 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:37.219+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:37.302 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:37.299+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:37.303 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:37.300+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:37.623 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:37.619+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:37.758 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:37.754+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:38.181 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:38.176+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:38.190 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:38.187+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:38.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:38.341+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:38.344 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:38.342+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:38.590 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:38.586+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:38.751 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:38.748+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:39.219 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:39.215+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:39.230 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:39.226+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:39.391 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:39.387+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:39.392 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:39.389+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:39.631 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:39.627+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:39.755 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:39.749+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:40.234 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:40.230+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:40.254 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:40.250+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:40.381 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:40.377+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:40.382 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:40.379+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:40.643 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:40.640+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:40.769 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:40.761+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:41.210 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:41.205+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:41.277 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:41.222+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:41.411 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:41.402+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:41.411 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:41.404+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:41.664 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:41.659+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:41.760 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:41.756+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:42.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:42.194+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:42.250 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:42.245+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:42.428 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:42.424+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:42.428 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:42.426+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:42.695 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:42.691+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:42.762 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:42.757+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:43.185 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:43.181+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:43.239 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:43.235+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:43.436 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:43.431+0000 1694f700 -1 osd.6 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:43.436 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:43.433+0000 1694f700 -1 osd.6 813 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:43.732 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:43.728+0000 1694f700 -1 osd.2 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:43.809 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:43.805+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:44.207 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:44.201+0000 1694f700 -1 osd.7 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:44.263 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:44.260+0000 1694f700 -1 osd.1 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:44.556 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:44.477+0000 1694f700 -1 osd.6 814 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:44.573 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:44.479+0000 1694f700 -1 osd.6 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:44.710 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:44.707+0000 1694f700 -1 osd.2 814 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:44.811 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:44.808+0000 1694f700 -1 osd.4 813 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:45.239 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:45.235+0000 1694f700 -1 osd.7 814 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:45.310 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:45.307+0000 1694f700 -1 osd.1 814 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:45.393 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:40:45.385+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e813 v813) 2019-12-12T00:40:45.756 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:45.713+0000 1694f700 -1 osd.6 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:45.757 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:45.715+0000 1694f700 -1 osd.6 815 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:45.879 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:45.826+0000 1694f700 -1 osd.2 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:46.161 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:46.065+0000 1694f700 -1 osd.4 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:46.237 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:46.233+0000 1694f700 -1 osd.7 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:46.276 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:46.271+0000 1694f700 -1 osd.1 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:46.709 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:46.704+0000 1694f700 -1 osd.6 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:46.710 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:46.706+0000 1694f700 -1 osd.6 815 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:46.790 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:46.786+0000 1694f700 -1 osd.2 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:47.139 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:47.135+0000 1694f700 -1 osd.4 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:47.198 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:47.193+0000 1694f700 -1 osd.7 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:47.317 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:47.308+0000 1694f700 -1 osd.1 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:47.950 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:47.837+0000 1694f700 -1 osd.2 816 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:48.035 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:47.785+0000 1694f700 -1 osd.6 816 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:48.095 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:47.929+0000 1694f700 -1 osd.6 816 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:48.184 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:48.180+0000 1694f700 -1 osd.4 815 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:48.351 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:48.195+0000 1694f700 -1 osd.7 816 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:48.391 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:48.384+0000 1694f700 -1 osd.1 816 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:48.968 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:48.962+0000 1694f700 -1 osd.6 817 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:48.981 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:48.964+0000 1694f700 -1 osd.6 817 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:49.041 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:49.019+0000 1694f700 -1 osd.2 817 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:49.195 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:49.192+0000 1694f700 -1 osd.4 816 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:49.326 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:49.262+0000 1694f700 -1 osd.7 817 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:49.401 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:49.387+0000 1694f700 -1 osd.1 817 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:50.028 INFO:tasks.ceph.osd.2.smithi079.stderr:2019-12-12T00:40:50.024+0000 1694f700 -1 osd.2 818 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.135430+0000 front 2019-12-12T00:39:26.276505+0000 (oldest deadline 2019-12-12T00:40:09.572699+0000) 2019-12-12T00:40:50.116 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:50.039+0000 1694f700 -1 osd.6 818 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.055327+0000 front 2019-12-12T00:39:26.136953+0000 (oldest deadline 2019-12-12T00:40:11.327475+0000) 2019-12-12T00:40:50.118 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:50.040+0000 1694f700 -1 osd.6 818 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:50.150 INFO:tasks.ceph.osd.4.smithi199.stderr:2019-12-12T00:40:50.147+0000 1694f700 -1 osd.4 817 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:26.688378+0000 front 2019-12-12T00:39:26.743100+0000 (oldest deadline 2019-12-12T00:40:11.939798+0000) 2019-12-12T00:40:50.405 INFO:tasks.ceph.osd.7.smithi199.stderr:2019-12-12T00:40:50.279+0000 1694f700 -1 osd.7 818 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:23.211060+0000 front 2019-12-12T00:39:23.233479+0000 (oldest deadline 2019-12-12T00:40:09.021148+0000) 2019-12-12T00:40:50.413 INFO:tasks.ceph.osd.1.smithi079.stderr:2019-12-12T00:40:50.400+0000 1694f700 -1 osd.1 818 heartbeat_check: no reply from 172.21.15.79:6806 osd.3 since back 2019-12-12T00:39:27.851717+0000 front 2019-12-12T00:39:27.854601+0000 (oldest deadline 2019-12-12T00:40:11.026542+0000) 2019-12-12T00:40:50.454 INFO:tasks.ceph.mon.a.smithi079.stderr:2019-12-12T00:40:50.445+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.79:6802/10236,v1:172.21.15.79:6803/10236] for 45sec e813 v813) 2019-12-12T00:40:51.773 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:51.674+0000 1694f700 -1 osd.6 819 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:53.206 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:53.178+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:54.150 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:54.149+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:54.245 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:08:41.176 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.256 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:08:41.161 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.258 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:08:41.190 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.261 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:08:41.193 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.261 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:08:41.166 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.267 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:08:41.198 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.481 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:08:41.385 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.545 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:08:41.477 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:54.546 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:08:41.477 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:40:55.201 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:55.199+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:56.194 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:56.192+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:57.203 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:57.201+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:58.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:40:58.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:40:58.218 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:58.216+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:40:59.197 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:40:59.195+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:00.224 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:00.223+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:01.193 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:01.190+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:02.224 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:02.221+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:03.266 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:03.263+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:04.220 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:04.217+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:05.223 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:05.220+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:06.218 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:06.215+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:07.185 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:07.183+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:08.205 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:08.203+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:09.247 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:09.244+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:10.255 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:10.252+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:11.235 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:11.232+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:12.403 INFO:tasks.ceph.osd.6.smithi199.stderr:2019-12-12T00:41:12.398+0000 1694f700 -1 osd.6 820 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.5065.0:30 7.7 7:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e805) 2019-12-12T00:41:20.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:41:20.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:41:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:41:42.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:41:43.450 INFO:tasks.ceph.osd.3.smithi079.stderr:*** Caught signal (Aborted) ** 2019-12-12T00:41:43.451 INFO:tasks.ceph.osd.3.smithi079.stderr: in thread 34f8d700 thread_name:tp_osd_tp 2019-12-12T00:41:43.527 INFO:tasks.ceph.osd.3.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:41:43.528 INFO:tasks.ceph.osd.3.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:41:43.528 INFO:tasks.ceph.osd.3.smithi079.stderr: 2: (_IO_vfprintf()+0x3fd4) [0xd15c584] 2019-12-12T00:41:43.528 INFO:tasks.ceph.osd.3.smithi079.stderr: 3: (__vsnprintf_chk()+0x95) [0xd225d75] 2019-12-12T00:41:43.528 INFO:tasks.ceph.osd.3.smithi079.stderr: 4: (cls_log()+0xfa) [0x69bf2a] 2019-12-12T00:41:43.528 INFO:tasks.ceph.osd.3.smithi079.stderr: 5: (()+0x23549) [0x209ec549] 2019-12-12T00:41:43.528 INFO:tasks.ceph.osd.3.smithi079.stderr: 6: (()+0x15342) [0x209de342] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 7: (()+0x65cdee) [0x764dee] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-12T00:41:43.529 INFO:tasks.ceph.osd.3.smithi079.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-12T00:41:43.530 INFO:tasks.ceph.osd.3.smithi079.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-12T00:41:43.530 INFO:tasks.ceph.osd.3.smithi079.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-12T00:41:43.530 INFO:tasks.ceph.osd.3.smithi079.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-12T00:41:43.530 INFO:tasks.ceph.osd.3.smithi079.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-12T00:41:43.530 INFO:tasks.ceph.osd.3.smithi079.stderr: 19: (()+0x7e65) [0xc096e65] 2019-12-12T00:41:43.530 INFO:tasks.ceph.osd.3.smithi079.stderr: 20: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:41:43.549 INFO:tasks.ceph.osd.3.smithi079.stderr:2019-12-12T00:41:43.544+0000 34f8d700 -1 *** Caught signal (Aborted) ** 2019-12-12T00:41:43.549 INFO:tasks.ceph.osd.3.smithi079.stderr: in thread 34f8d700 thread_name:tp_osd_tp 2019-12-12T00:41:43.549 INFO:tasks.ceph.osd.3.smithi079.stderr: 2019-12-12T00:41:43.549 INFO:tasks.ceph.osd.3.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:41:43.549 INFO:tasks.ceph.osd.3.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 2: (_IO_vfprintf()+0x3fd4) [0xd15c584] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 3: (__vsnprintf_chk()+0x95) [0xd225d75] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 4: (cls_log()+0xfa) [0x69bf2a] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 5: (()+0x23549) [0x209ec549] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 6: (()+0x15342) [0x209de342] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 7: (()+0x65cdee) [0x764dee] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-12T00:41:43.550 INFO:tasks.ceph.osd.3.smithi079.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-12T00:41:43.551 INFO:tasks.ceph.osd.3.smithi079.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-12T00:41:43.551 INFO:tasks.ceph.osd.3.smithi079.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-12T00:41:43.551 INFO:tasks.ceph.osd.3.smithi079.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-12T00:41:43.551 INFO:tasks.ceph.osd.3.smithi079.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-12T00:41:43.551 INFO:tasks.ceph.osd.3.smithi079.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-12T00:41:43.551 INFO:tasks.ceph.osd.3.smithi079.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-12T00:41:43.551 INFO:tasks.ceph.osd.3.smithi079.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-12T00:41:43.552 INFO:tasks.ceph.osd.3.smithi079.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-12T00:41:43.552 INFO:tasks.ceph.osd.3.smithi079.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-12T00:41:43.552 INFO:tasks.ceph.osd.3.smithi079.stderr: 19: (()+0x7e65) [0xc096e65] 2019-12-12T00:41:43.552 INFO:tasks.ceph.osd.3.smithi079.stderr: 20: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:41:43.552 INFO:tasks.ceph.osd.3.smithi079.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-12T00:41:43.552 INFO:tasks.ceph.osd.3.smithi079.stderr: 2019-12-12T00:41:43.904 INFO:tasks.ceph.osd.3.smithi079.stderr: 0> 2019-12-12T00:41:43.544+0000 34f8d700 -1 *** Caught signal (Aborted) ** 2019-12-12T00:41:43.905 INFO:tasks.ceph.osd.3.smithi079.stderr: in thread 34f8d700 thread_name:tp_osd_tp 2019-12-12T00:41:43.905 INFO:tasks.ceph.osd.3.smithi079.stderr: 2019-12-12T00:41:43.905 INFO:tasks.ceph.osd.3.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:41:43.905 INFO:tasks.ceph.osd.3.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:41:43.905 INFO:tasks.ceph.osd.3.smithi079.stderr: 2: (_IO_vfprintf()+0x3fd4) [0xd15c584] 2019-12-12T00:41:43.905 INFO:tasks.ceph.osd.3.smithi079.stderr: 3: (__vsnprintf_chk()+0x95) [0xd225d75] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 4: (cls_log()+0xfa) [0x69bf2a] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 5: (()+0x23549) [0x209ec549] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 6: (()+0x15342) [0x209de342] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 7: (()+0x65cdee) [0x764dee] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-12T00:41:43.906 INFO:tasks.ceph.osd.3.smithi079.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-12T00:41:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-12T00:41:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-12T00:41:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-12T00:41:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-12T00:41:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-12T00:41:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-12T00:41:43.907 INFO:tasks.ceph.osd.3.smithi079.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-12T00:41:43.908 INFO:tasks.ceph.osd.3.smithi079.stderr: 19: (()+0x7e65) [0xc096e65] 2019-12-12T00:41:43.908 INFO:tasks.ceph.osd.3.smithi079.stderr: 20: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:41:43.908 INFO:tasks.ceph.osd.3.smithi079.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-12T00:41:43.908 INFO:tasks.ceph.osd.3.smithi079.stderr: 2019-12-12T00:41:44.326 INFO:tasks.ceph.osd.3.smithi079.stderr: -6> 2019-12-12T00:41:43.544+0000 34f8d700 -1 *** Caught signal (Aborted) ** 2019-12-12T00:41:44.326 INFO:tasks.ceph.osd.3.smithi079.stderr: in thread 34f8d700 thread_name:tp_osd_tp 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: ceph version 15.0.0-7938-g1b78fd0 (1b78fd04fb1980223b57f0a3f644592e4e26b518) octopus (dev) 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: 2: (_IO_vfprintf()+0x3fd4) [0xd15c584] 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: 3: (__vsnprintf_chk()+0x95) [0xd225d75] 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: 4: (cls_log()+0xfa) [0x69bf2a] 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: 5: (()+0x23549) [0x209ec549] 2019-12-12T00:41:44.327 INFO:tasks.ceph.osd.3.smithi079.stderr: 6: (()+0x15342) [0x209de342] 2019-12-12T00:41:44.328 INFO:tasks.ceph.osd.3.smithi079.stderr: 7: (()+0x65cdee) [0x764dee] 2019-12-12T00:41:44.328 INFO:tasks.ceph.osd.3.smithi079.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-12T00:41:44.328 INFO:tasks.ceph.osd.3.smithi079.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-12T00:41:44.328 INFO:tasks.ceph.osd.3.smithi079.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-12T00:41:44.328 INFO:tasks.ceph.osd.3.smithi079.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-12T00:41:44.328 INFO:tasks.ceph.osd.3.smithi079.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-12T00:41:44.328 INFO:tasks.ceph.osd.3.smithi079.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-12T00:41:44.329 INFO:tasks.ceph.osd.3.smithi079.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-12T00:41:44.329 INFO:tasks.ceph.osd.3.smithi079.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-12T00:41:44.329 INFO:tasks.ceph.osd.3.smithi079.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-12T00:41:44.329 INFO:tasks.ceph.osd.3.smithi079.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-12T00:41:44.329 INFO:tasks.ceph.osd.3.smithi079.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-12T00:41:44.329 INFO:tasks.ceph.osd.3.smithi079.stderr: 19: (()+0x7e65) [0xc096e65] 2019-12-12T00:41:44.329 INFO:tasks.ceph.osd.3.smithi079.stderr: 20: (clone()+0x6d) [0xd20e88d] 2019-12-12T00:41:44.330 INFO:tasks.ceph.osd.3.smithi079.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-12T00:41:44.330 INFO:tasks.ceph.osd.3.smithi079.stderr: 2019-12-12T00:41:48.845 INFO:tasks.ceph.osd.3.smithi079.stderr:daemon-helper: command crashed with signal 6 2019-12-12T00:42:04.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:42:04.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:42:26.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:42:26.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:42:48.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:42:48.159+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:43:10.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:43:10.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:43:32.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:43:32.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:43:54.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:43:54.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:44:16.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:44:16.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:44:38.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:44:38.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:45:00.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:45:00.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:45:22.162 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:45:22.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:45:44.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:45:44.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:45:50.743 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:13:37.675 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:50.750 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:13:37.682 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:50.764 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:13:37.695 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:50.768 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:13:37.700 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:50.774 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:13:37.678 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:50.794 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:13:37.698 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:51.007 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:13:37.938 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:51.022 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:13:37.954 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:45:51.060 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:13:37.964 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:46:06.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:46:06.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:46:28.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:46:28.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:46:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:46:50.160+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:47:07.103 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:14:54.035 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.105 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:14:54.037 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.154 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:14:54.085 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.158 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:14:54.089 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.185 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:14:54.116 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.185 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:14:54.088 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.193 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:14:54.097 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.216 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:14:54.147 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:07.304 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:14:54.206 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:47:12.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:47:12.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:47:34.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:47:34.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:47:56.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:47:56.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:48:18.163 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:48:18.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:48:40.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:48:40.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:49:02.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:49:02.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:49:24.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:49:24.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:49:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:49:46.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:50:08.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:50:08.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:50:30.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:50:30.161+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:50:52.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:50:52.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:50:59.163 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:18:46.094 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.166 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:18:46.098 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.187 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:18:46.090 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.201 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:18:46.105 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.271 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:18:46.202 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.280 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:18:46.211 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.385 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:18:46.316 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.418 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:18:46.322 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:50:59.481 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:18:46.409 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:51:14.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:51:14.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:51:36.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:51:36.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:51:58.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:51:58.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:52:02.660 INFO:tasks.rgw.client.0.smithi079.stdout:2019-12-12T00:52:02.657+0000 20e25700 -1 RGWWatcher::handle_error cookie 5433917920 err (110) Connection timed out 2019-12-12T00:52:20.164 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:52:20.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:52:42.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:52:42.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:53:04.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:53:04.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:53:26.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:53:26.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:53:37.305 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:21:24.237 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.316 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:21:24.247 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.330 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:21:24.261 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.337 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:21:24.268 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.342 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:21:24.246 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.391 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:21:24.295 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.679 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:21:24.610 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.760 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:21:24.663 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:37.785 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:21:24.713 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:53:48.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:53:48.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:54:10.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:54:10.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:54:30.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:54:30.172+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] conn(0x55c1c3f90390 msgr2=0x55c1c3f928c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T00:54:32.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:54:32.162+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:54:40.219 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:54:40.217+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c034ed0 msgr2=0x7fd98c0352e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T00:54:49.919 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:54:49.916+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c1c3f7d9a0 msgr2=0x55c1c3f843c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T00:54:54.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:54:54.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:55:16.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:55:16.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:55:29.260 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:23:16.191 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.265 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:23:16.197 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.324 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:23:16.256 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.329 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:23:16.261 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.374 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:23:16.278 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.377 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:23:16.281 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.392 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:23:16.324 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.410 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:23:16.342 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:29.515 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:23:16.419 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.328 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:23:21.260 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.356 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:23:21.288 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.363 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:23:21.294 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.388 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:23:21.320 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.407 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:23:21.338 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.412 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:23:21.316 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.422 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:23:21.325 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.424 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:23:21.355 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:34.448 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:23:21.352 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:55:38.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:55:38.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:56:00.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:56:00.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:56:22.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:56:22.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:56:44.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:56:44.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:57:06.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:57:06.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:57:28.165 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:57:28.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:57:40.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:57:40.206+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c03b9a0 msgr2=0x7fd98c03bdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T00:57:50.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:57:50.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:58:12.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:58:12.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:58:34.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:58:34.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:58:49.602 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:26:36.534 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.609 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:26:36.541 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.627 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:26:36.557 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.631 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:26:36.562 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.637 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:26:36.541 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.654 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:26:36.557 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.769 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:26:36.701 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.818 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:26:36.745 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:49.893 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:26:36.796 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T00:58:56.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:58:56.163+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:59:18.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:59:18.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T00:59:40.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T00:59:40.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:00:02.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:00:02.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:00:24.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:00:24.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:00:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:00:46.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:01:08.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:01:08.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:01:30.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:01:30.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:01:52.166 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:01:52.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:02:14.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:02:14.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:02:36.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:02:36.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:02:58.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:02:58.164+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:03:20.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:03:20.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:03:42.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:03:42.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:03:56.047 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:31:42.979 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.054 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:31:42.986 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.116 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:31:43.048 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.122 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:31:43.053 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.142 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:31:43.046 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.145 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:31:43.049 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.308 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:31:43.239 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.354 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:31:43.286 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:56.425 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:31:43.328 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.157 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:31:45.088 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.165 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:31:45.097 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.186 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:31:45.117 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.334 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:31:45.238 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.446 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:31:45.240 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.446 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:31:45.263 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.509 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:31:45.440 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.510 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:31:45.442 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:03:58.520 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:31:45.452 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:04:04.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:04:04.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:04:26.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:04:26.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:04:48.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:04:48.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:05:10.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:05:10.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:05:32.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:05:32.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:05:54.167 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:05:54.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:06:16.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:06:16.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:06:38.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:06:38.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:07:00.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:07:00.165+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:07:22.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:07:22.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:07:44.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:07:44.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:08:06.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:08:06.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:08:28.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:08:28.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:08:50.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:08:50.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:09:12.097 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:36:59.029 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.108 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:36:59.040 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.115 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:36:59.046 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.118 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:36:59.050 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.125 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:36:59.028 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.135 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:36:59.038 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:09:12.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:09:12.329 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:36:59.261 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.332 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:36:59.264 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:12.375 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:36:59.279 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:09:34.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:09:34.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:09:56.168 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:09:56.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:10:18.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:10:18.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:10:40.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:10:40.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:11:02.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:11:02.166+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:11:24.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:11:24.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:11:46.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:11:46.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:12:08.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:12:08.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:12:17.831 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:40:04.763 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:17.838 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:40:04.770 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:17.888 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:40:04.819 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:17.889 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:40:04.821 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:17.910 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:40:04.813 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:17.928 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:40:04.831 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:17.992 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:40:04.924 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:18.055 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:40:04.986 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:18.089 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:40:04.993 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:12:30.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:12:30.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:12:52.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:12:52.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:13:14.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:13:14.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:13:36.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:13:36.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:13:58.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:13:58.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:14:00.401 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:14:00.398+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c009a20 msgr2=0x7fd98c00be40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T01:14:20.169 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:14:20.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:14:23.531 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:42:10.463 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.540 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:42:10.472 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.554 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:42:10.458 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.555 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:42:10.487 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.561 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:42:10.493 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.562 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:42:10.466 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.664 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:42:10.596 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.683 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:42:10.586 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:23.693 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:42:10.624 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:14:42.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:14:42.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:15:04.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:15:04.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:15:26.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:15:26.167+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:15:48.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:15:48.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:16:10.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:16:10.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:16:32.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:16:32.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:16:54.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:16:54.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:17:16.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:17:16.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:17:38.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:17:38.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:18:00.170 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:18:00.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:18:22.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:18:22.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:18:44.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:18:44.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:19:06.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:19:06.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:19:28.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:19:28.168+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:19:34.483 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:47:21.414 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.496 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:47:21.428 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.506 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:47:21.437 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.510 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:47:21.442 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.518 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:47:21.421 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.526 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:47:21.430 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.699 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:47:21.630 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.729 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:47:21.660 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:34.758 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:47:21.661 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:19:50.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:19:50.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:20:12.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:20:12.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:20:34.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:20:34.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:20:39.650 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:48:26.581 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.715 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:48:26.646 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.735 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:48:26.638 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.739 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:48:26.642 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.743 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:48:26.675 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.752 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:48:26.684 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.786 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:48:26.717 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.811 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:48:26.743 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:39.824 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:48:26.727 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:20:56.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:20:56.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:21:18.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:21:18.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:21:40.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:21:40.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:22:02.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:22:02.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:22:24.171 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:22:24.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:22:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:22:46.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:23:08.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:23:08.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:23:30.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:23:30.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:23:52.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:23:52.169+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:24:14.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:24:14.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:24:36.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:24:36.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:24:45.549 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:52:32.480 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.557 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:52:32.489 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.572 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:52:32.475 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.577 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:52:32.509 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.584 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:52:32.487 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.588 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:52:32.519 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.748 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:52:32.679 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.764 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:52:32.695 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:45.825 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:52:32.728 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:24:58.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:24:58.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:25:20.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:25:20.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:25:42.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:25:42.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:26:04.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:26:04.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:26:26.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:26:26.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:26:48.172 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:26:48.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:27:10.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:27:10.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:27:32.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:27:32.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:27:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:27:45.526+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] conn(0x7fd98c0305a0 msgr2=0x7fd98c0309b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T01:27:54.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:27:54.170+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:28:16.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:28:16.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:28:38.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:28:38.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:29:00.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:29:00.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:29:01.314 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:56:48.239 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.316 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:56:48.241 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.361 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:56:48.285 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.363 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:56:48.286 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.393 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:56:48.291 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.403 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:56:48.300 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.482 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:56:48.407 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.506 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:56:48.432 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:01.539 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:56:48.435 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:22.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:29:22.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:29:44.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:29:44.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:29:57.565 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:57:44.497 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.576 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:57:44.508 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.583 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:57:44.515 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.586 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:57:44.490 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.587 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:57:44.519 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.591 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:57:44.495 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.655 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:08:57:44.559 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.664 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:08:57:44.596 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:29:57.678 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:08:57:44.610 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:30:06.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:30:06.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:30:28.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:30:28.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:30:50.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:30:50.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:31:12.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:31:12.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:31:30.568 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:31:30.566+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c0305a0 msgr2=0x7fd98c0309b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T01:31:34.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:31:34.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:31:56.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:31:56.171+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:32:18.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:32:18.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:32:40.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:32:40.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:33:02.173 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:33:02.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:33:24.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:33:24.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:33:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:33:46.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:34:08.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:34:08.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:34:30.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:34:30.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:34:52.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:34:52.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:35:09.225 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:02:56.157 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.234 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:02:56.167 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.250 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:02:56.182 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.254 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:02:56.186 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.257 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:02:56.160 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.271 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:02:56.175 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.464 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:02:56.397 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.475 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:02:56.407 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:09.556 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:02:56.460 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:35:14.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:35:14.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:35:36.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:35:36.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:35:58.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:35:58.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:36:18.049 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:04:04.981 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.054 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:04:04.986 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.113 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:04:05.045 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.117 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:04:05.049 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.156 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:04:05.089 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.172 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:04:05.076 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.176 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:04:05.080 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.188 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:04:05.120 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:18.271 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:04:05.175 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:36:20.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:36:20.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:36:42.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:36:42.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:37:04.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:37:04.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:37:23.405 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:05:10.338 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.410 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:05:10.343 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.424 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:05:10.357 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.435 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:05:10.339 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.448 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:05:10.381 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.462 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:05:10.395 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.482 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:05:10.386 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.521 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:05:10.453 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:23.538 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:05:10.442 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:37:26.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:37:26.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:37:48.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:37:48.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:37:55.656 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:37:55.654+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] conn(0x7fd98c0305a0 msgr2=0x7fd98c0309b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T01:38:10.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:38:10.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:38:32.174 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:38:32.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:38:54.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:38:54.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:39:16.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:39:16.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:39:38.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:39:38.172+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:40:00.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:40:00.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:40:19.044 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:08:05.976 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.046 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:08:05.979 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.056 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:08:05.989 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.061 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:08:05.994 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.064 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:08:05.968 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.069 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:08:05.973 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.172 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:08:06.104 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.235 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:08:06.138 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:19.250 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:08:06.182 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:40:22.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:40:22.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:40:44.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:40:44.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:41:06.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:41:06.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:41:28.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:41:28.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:41:50.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:41:50.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:42:12.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:42:12.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:42:34.175 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:42:34.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:42:56.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:42:56.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:43:18.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:43:18.173+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:43:40.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:43:40.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:44:02.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:44:02.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:44:24.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:44:24.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:44:46.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:44:46.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:45:08.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:45:08.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:45:30.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:45:30.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:45:33.525 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:13:20.458 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.529 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:13:20.462 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.539 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:13:20.472 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.544 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:13:20.477 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.547 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:13:20.451 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.572 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:13:20.476 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.730 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:13:20.663 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.739 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:13:20.672 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:33.841 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:13:20.745 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.170 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:13:37.103 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.210 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:13:37.143 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.243 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:13:37.175 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.247 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:13:37.180 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.253 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:13:37.157 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.253 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:13:37.186 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.259 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:13:37.164 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.279 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:13:37.211 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:50.298 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:13:37.202 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:45:52.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:45:52.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:46:14.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:46:14.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:46:36.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:46:36.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:46:58.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:46:58.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:47:20.176 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:47:20.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:47:42.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:47:42.174+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:48:04.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:48:04.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:48:26.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:48:26.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:48:48.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:48:48.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:49:10.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:49:10.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:49:32.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:49:32.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:49:54.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:49:54.175+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:50:16.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:50:16.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:50:38.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:50:38.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:50:43.848 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:18:30.781 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:43.863 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:18:30.796 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:43.873 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:18:30.777 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:43.877 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:18:30.809 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:43.881 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:18:30.786 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:43.884 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:18:30.817 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:44.043 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:18:30.976 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:44.093 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:18:31.026 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:50:44.143 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:18:31.047 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:51:00.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:51:00.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:51:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:51:00.817+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c036c40 msgr2=0x7fd98c00ba30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T01:51:22.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:51:22.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:51:44.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:51:44.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:52:06.177 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:52:06.176+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:52:28.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:52:28.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:52:50.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:52:50.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:53:12.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:53:12.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:53:34.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:53:34.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:53:56.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:53:56.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:54:12.048 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:21:58.981 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.057 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:21:58.990 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.109 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:21:59.012 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.110 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:21:59.015 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.130 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:21:59.063 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.133 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:21:59.066 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.205 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:21:59.138 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.216 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:21:59.120 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:12.246 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:21:59.179 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:54:18.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:54:18.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:54:40.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:54:40.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:55:02.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:55:02.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:55:24.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:55:24.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:55:46.178 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:55:46.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:55:55.088 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:23:42.022 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.102 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:23:42.035 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.104 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:23:42.037 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.105 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:23:42.038 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.105 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:23:42.009 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.118 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:23:42.023 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.187 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:23:42.120 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.189 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:23:42.123 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T01:55:55.214 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:23:42.119 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T01:56:08.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:56:08.177+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:56:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:56:30.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:56:52.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:56:52.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:57:14.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:57:14.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:57:36.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:57:36.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:57:58.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:57:58.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:58:20.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:58:20.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:58:42.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:58:42.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:59:04.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:59:04.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:59:26.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:59:26.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T01:59:48.179 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T01:59:48.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:00:10.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:00:10.178+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:00:32.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:00:32.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:00:54.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:00:54.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:01:04.595 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:28:51.528 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.603 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:28:51.536 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.612 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:28:51.545 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.617 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:28:51.550 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.624 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:28:51.528 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.633 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:28:51.537 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.876 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:28:51.780 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.943 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:28:51.877 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:04.966 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:28:51.899 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:01:16.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:01:16.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:01:38.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:01:38.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:02:00.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:02:00.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:02:22.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:02:22.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:02:33.792 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:30:20.725 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.794 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:30:20.727 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.815 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:30:20.719 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.863 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:30:20.796 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.868 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:30:20.801 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.880 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:30:20.813 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.885 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:30:20.790 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.934 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:30:20.867 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:33.941 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:30:20.845 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:02:44.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:02:44.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:03:06.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:03:06.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:03:28.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:03:28.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:03:50.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:03:50.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:04:12.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:04:12.179+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:04:34.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:04:34.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:04:56.180 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:04:56.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:05:18.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:05:18.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:05:40.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:05:40.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:06:02.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:06:02.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:06:16.539 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:34:03.472 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.551 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:34:03.484 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.555 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:34:03.488 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.557 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:34:03.490 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.563 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:34:03.468 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.567 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:34:03.472 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.711 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:34:03.644 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.713 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:34:03.647 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:16.760 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:34:03.665 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:06:24.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:06:24.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:06:46.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:06:46.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:07:08.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:07:08.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:07:30.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:07:30.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:07:52.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:07:52.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:08:14.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:08:14.180+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:08:36.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:08:36.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:08:58.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:08:58.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:09:20.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:09:20.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:09:42.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:09:42.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:10:04.181 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:10:04.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:10:26.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:10:26.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:10:48.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:10:48.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:10:55.570 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:38:42.504 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.581 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:38:42.514 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.603 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:38:42.537 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.648 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:38:42.552 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.651 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:38:42.555 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.657 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:38:42.590 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.758 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:38:42.692 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.791 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:38:42.724 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:10:55.833 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:38:42.737 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:10.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:11:10.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:11:26.878 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:39:13.812 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:26.891 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:39:13.824 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:26.896 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:39:13.829 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:26.904 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:39:13.837 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:26.904 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:39:13.809 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:26.916 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:39:13.820 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:26.945 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:39:13.879 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:26.959 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:39:13.892 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:27.012 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:39:13.915 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:11:32.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:11:32.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:11:54.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:11:54.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:12:16.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:12:16.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:12:38.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:12:38.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:13:00.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:13:00.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:13:22.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:13:22.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:13:44.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:13:44.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:14:06.182 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:14:06.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:14:28.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:14:28.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:14:50.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:14:50.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:15:12.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:15:12.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:15:34.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:15:34.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:15:56.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:15:56.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:16:18.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:16:18.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:16:38.926 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:44:25.860 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:38.935 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:44:25.869 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:38.943 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:44:25.875 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:38.954 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:44:25.887 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:38.954 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:44:25.859 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:38.960 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:44:25.864 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:39.200 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:44:26.133 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:39.231 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:44:26.135 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:39.248 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:44:26.182 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:16:40.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:16:40.181+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:17:02.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:17:02.183+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:17:07.132 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:44:54.065 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.146 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:44:54.079 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.186 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:44:54.119 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.228 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:44:54.161 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.237 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:44:54.170 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.255 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:44:54.159 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.260 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:44:54.164 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.274 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:44:54.207 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:07.317 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:44:54.221 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:17:24.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:17:24.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:17:46.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:17:46.183+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:18:08.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:18:08.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:18:30.183 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:18:30.183+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:18:52.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:18:52.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:19:14.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:19:14.183+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:19:17.613 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:47:04.546 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.657 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:47:04.591 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.671 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:47:04.603 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.676 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:47:04.609 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.687 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:47:04.591 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.690 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:47:04.595 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.769 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:47:04.702 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.783 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:47:04.716 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:17.822 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:47:04.726 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:19:26.099 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:19:26.098+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c03bee0 msgr2=0x7fd98c0362f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T02:19:36.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:19:36.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:19:58.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:19:58.183+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:20:20.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:20:20.182+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:20:42.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:20:42.183+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:21:04.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:21:04.183+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:21:26.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:21:26.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:21:48.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:21:48.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:21:51.216 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:49:38.149 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.220 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:49:38.153 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.227 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:49:38.160 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.230 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:49:38.134 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.231 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:49:38.164 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.235 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:49:38.139 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.338 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:49:38.271 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.350 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:49:38.278 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:21:51.399 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:49:38.303 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:22:10.184 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:22:10.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:22:32.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:22:32.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:22:54.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:22:54.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:23:16.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:23:16.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:23:38.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:23:38.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:24:00.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:24:00.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:24:22.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:24:22.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:24:44.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:24:44.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:25:06.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:25:06.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:25:28.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:25:28.184+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:25:50.185 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:25:50.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:26:12.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:26:12.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:26:34.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:26:34.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:26:56.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:26:56.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:27:04.425 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:54:51.358 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.441 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:54:51.374 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.452 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:54:51.385 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.456 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:54:51.389 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.472 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:54:51.376 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.483 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:54:51.387 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.648 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:54:51.581 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.661 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:54:51.593 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:04.750 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:54:51.654 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:18.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:27:18.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:27:39.570 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:55:26.503 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.581 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:55:26.485 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.582 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:55:26.515 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.583 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:55:26.487 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.617 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:09:55:26.521 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.618 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:55:26.551 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.630 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:09:55:26.564 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.647 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:55:26.580 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:39.678 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:09:55:26.611 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:27:40.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:27:40.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:28:02.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:28:02.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:28:24.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:28:24.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:28:46.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:28:46.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:29:08.186 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:29:08.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:29:30.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:29:30.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:29:52.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:29:52.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:30:14.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:30:14.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:30:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:30:36.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:30:58.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:30:58.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:31:20.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:31:20.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:31:42.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:31:42.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:32:04.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:32:04.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:32:15.545 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:00:02.478 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.553 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:00:02.487 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.562 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:00:02.494 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.565 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:00:02.497 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.571 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:00:02.475 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.577 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:00:02.481 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.760 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:00:02.693 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.841 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:00:02.774 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:15.857 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:00:02.761 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:32:26.187 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:32:26.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:32:48.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:32:48.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:33:10.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:33:10.185+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:33:32.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:33:32.187+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:33:54.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:33:54.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:34:16.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:34:16.187+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:34:38.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:34:38.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:35:00.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:35:00.187+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:35:22.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:35:22.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:35:44.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:35:44.187+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:36:01.323 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:03:48.256 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.370 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:03:48.303 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.401 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:03:48.305 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.404 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:03:48.308 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.437 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:03:48.370 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.444 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:03:48.376 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.536 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:03:48.467 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.536 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:03:48.441 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:01.616 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:03:48.549 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:36:06.188 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:36:06.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:36:28.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:36:28.187+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:36:50.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:36:50.186+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:36:56.297 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:36:56.296+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c024b40 msgr2=0x7fd98c03b460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T02:37:12.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:37:12.187+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:37:25.911 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:05:12.844 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.506 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:05:12.852 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.506 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:05:12.860 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.506 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:05:13.034 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.507 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:05:12.854 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.507 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:05:12.964 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.507 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:05:12.859 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.507 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:05:12.864 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:26.507 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:05:12.967 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:37:34.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:37:34.187+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:37:56.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:37:56.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:38:18.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:38:18.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:38:40.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:38:40.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:39:02.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:39:02.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:39:24.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:39:24.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:39:46.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:39:46.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:39:51.306 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:39:51.305+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c03bee0 msgr2=0x7fd98c0362f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T02:40:08.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:40:08.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:40:30.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:40:30.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:40:52.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:40:52.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:41:14.189 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:41:14.188+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:41:36.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:41:36.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:41:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:41:58.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:42:20.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:42:20.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:42:37.945 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:10:24.878 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:37.948 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:10:24.881 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:37.963 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:10:24.896 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:37.966 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:10:24.871 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:37.968 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:10:24.901 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:37.977 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:10:24.882 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:38.155 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:10:25.088 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:38.190 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:10:25.123 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:38.273 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:10:25.177 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:42:42.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:42:42.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:43:04.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:43:04.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:43:26.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:43:26.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:43:48.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:43:48.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:44:10.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:44:10.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:44:23.319 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:12:10.253 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.326 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:12:10.260 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.344 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:12:10.276 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.347 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:12:10.278 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.374 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:12:10.279 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.378 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:12:10.283 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.388 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:12:10.322 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.432 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:12:10.363 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:23.443 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:12:10.348 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:44:32.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:44:32.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:44:54.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:44:54.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:45:16.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:45:16.189+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:45:38.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:45:38.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:46:00.190 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:46:00.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:46:22.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:46:22.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:46:44.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:46:44.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:47:06.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:47:06.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:47:28.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:47:28.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:47:50.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:47:50.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:47:50.277 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:15:37.210 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.288 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:15:37.222 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.294 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:15:37.227 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.300 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:15:37.233 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.442 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:15:37.346 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.442 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:15:37.375 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.451 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:15:37.355 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.460 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:15:37.394 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:47:50.805 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:15:37.709 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:48:12.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:48:12.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:48:34.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:48:34.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:48:56.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:48:56.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:49:18.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:49:18.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:49:40.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:49:40.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:50:02.191 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:50:02.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:50:24.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:50:24.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:50:46.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:50:46.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:51:08.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:51:08.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:51:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:51:30.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:51:52.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:51:52.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:52:14.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:52:14.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:52:36.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:52:36.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:52:50.171 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:20:37.104 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.220 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:20:37.153 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.223 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:20:37.156 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.251 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:20:37.184 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.280 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:20:37.184 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.284 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:20:37.188 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.415 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:20:37.347 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.434 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:20:37.367 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:50.495 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:20:37.399 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:52:58.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:52:58.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:53:04.420 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:20:51.353 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.429 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:20:51.362 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.435 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:20:51.367 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.439 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:20:51.372 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.450 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:20:51.355 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.464 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:20:51.397 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.471 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:20:51.404 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.476 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:20:51.381 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:04.542 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:20:51.447 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:53:20.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:53:20.190+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:53:42.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:53:42.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:54:04.192 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:54:04.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:54:26.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:54:26.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:54:48.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:54:48.191+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:55:10.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:55:10.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:55:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:55:32.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:55:54.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:55:54.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:56:16.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:56:16.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:56:38.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:56:38.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:57:00.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:57:00.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:57:22.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:57:22.192+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:57:36.688 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:25:23.621 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:36.731 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:25:23.664 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:36.757 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:25:23.690 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:36.762 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:25:23.695 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:36.817 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:25:23.722 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:36.822 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:25:23.726 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:36.934 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:25:23.867 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:37.056 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:25:23.990 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:37.095 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:25:23.999 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:57:44.193 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:57:44.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:58:06.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:58:06.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:58:14.401 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:26:01.334 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.415 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:26:01.349 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.424 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:26:01.357 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.433 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:26:01.366 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.433 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:26:01.338 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.456 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:26:01.361 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.484 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:26:01.417 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.494 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:26:01.426 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:14.550 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:26:01.454 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T02:58:28.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:58:28.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:58:50.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:58:50.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:59:12.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:59:12.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:59:34.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:59:34.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T02:59:56.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T02:59:56.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:00:18.194 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:00:18.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:00:40.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:00:40.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:01:02.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:01:02.193+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:01:12.415 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:28:59.348 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.454 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:28:59.387 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.491 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:28:59.424 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.493 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:28:59.426 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.533 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:28:59.437 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.540 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:28:59.444 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.594 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:28:59.527 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.662 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:28:59.594 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:12.686 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:28:59.590 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:01:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:01:24.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:01:46.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:01:46.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:02:08.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:02:08.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:02:30.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:02:30.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:02:52.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:02:52.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:03:14.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:03:14.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:03:24.379 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:31:11.312 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.384 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:31:11.318 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.395 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:31:11.328 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.398 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:31:11.302 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.399 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:31:11.332 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.413 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:31:11.318 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.487 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:31:11.420 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.511 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:31:11.444 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:24.564 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:31:11.469 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:03:36.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:03:36.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:03:58.195 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:03:58.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:04:05.816 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:04:05.815+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/250202870 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd8e4028b40 msgr2=0x7fd8e4022f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T03:04:20.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:04:20.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:04:41.574 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:04:41.572+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.79:6818/10237,v1:172.21.15.79:6819/10237] conn(0x7fd98c00fa20 msgr2=0x7fd98c00fe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:04:42.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:04:42.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:05:04.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:05:04.195+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:05:21.604 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:05:21.603+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c024b40 msgr2=0x7fd98c03b460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:05:26.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:05:26.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:05:48.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:05:48.195+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:06:10.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:06:10.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:06:32.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:06:32.195+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:06:54.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:06:54.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:07:16.196 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:07:16.195+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:07:38.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:07:38.194+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:08:00.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:08:00.195+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:08:22.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:08:22.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:08:37.187 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:36:24.120 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.193 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:36:24.126 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.204 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:36:24.136 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.211 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:36:24.144 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.221 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:36:24.124 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.253 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:36:24.138 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.426 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:36:24.359 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.453 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:36:24.386 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:37.487 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:36:24.391 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:08:44.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:08:44.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:09:06.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:09:06.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:09:28.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:09:28.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:09:34.520 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:37:21.453 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.528 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:37:21.460 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.528 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:37:21.462 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.558 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:37:21.491 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.588 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:37:21.521 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.596 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:37:21.529 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.615 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:37:21.519 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.620 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:37:21.525 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:34.677 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:37:21.582 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:09:50.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:09:50.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:10:12.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:10:12.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:10:34.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:10:34.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:10:56.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:10:56.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:11:18.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:11:18.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:11:40.197 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:11:40.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:11:46.631 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:11:46.630+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6800/10148,v1:172.21.15.199:6801/10148] conn(0x7fd9a4008600 msgr2=0x7fd9a4008a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T03:12:02.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:12:02.196+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:12:24.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:12:24.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:12:46.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:12:46.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:13:08.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:13:08.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:13:30.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:13:30.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:13:50.249 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:41:37.182 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.258 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:41:37.191 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.270 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:41:37.203 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.274 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:41:37.206 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.283 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:41:37.187 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.289 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:41:37.194 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.440 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:41:37.373 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.444 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:41:37.377 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:50.559 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:41:37.463 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:13:52.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:13:52.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:14:14.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:14:14.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:14:36.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:14:36.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:14:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:14:58.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:15:20.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:15:20.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:15:42.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:15:42.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:16:04.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:16:04.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:16:26.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:16:26.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:16:44.294 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:16:44.292+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/250202870 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x55c1c3fb95a0 msgr2=0x55c1c3fbbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:16:48.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:16:48.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:17:10.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:17:10.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:17:24.294 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:17:24.292+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/250202870 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fd8e4028f50 msgr2=0x7fd8e4036f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:17:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:17:32.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:17:54.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:17:54.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:17:56.079 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:45:43.012 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.086 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:45:43.019 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.127 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:45:43.059 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.129 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:45:43.061 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.159 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:45:43.063 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.162 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:45:43.066 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.207 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:45:43.140 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.267 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:45:43.199 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:17:56.327 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:45:43.231 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:18:16.199 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:18:16.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:18:38.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:18:38.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:19:00.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:19:00.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:19:02.447 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:46:49.379 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.455 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:46:49.388 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.462 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:46:49.395 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.467 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:46:49.399 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.480 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:46:49.384 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.505 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:46:49.409 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.530 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:46:49.462 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.532 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:46:49.465 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:02.659 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:46:49.564 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:19:22.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:19:22.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:19:44.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:19:44.197+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:20:06.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:20:06.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:20:28.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:20:28.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:20:50.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:20:50.199+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:21:12.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:21:12.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:21:34.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:21:34.199+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:21:56.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:21:56.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:22:18.200 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:22:18.199+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:22:40.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:22:40.198+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:23:02.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:23:02.199+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:23:24.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:23:24.199+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:23:46.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:23:46.199+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:24:08.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:24:08.199+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:24:14.905 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:52:01.838 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:14.916 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:52:01.848 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:14.941 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:52:01.845 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:14.951 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:52:01.883 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:14.962 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:52:01.894 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:14.967 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:52:01.871 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:15.176 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:52:02.108 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:15.202 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:52:02.135 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:15.209 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:52:02.113 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:24:30.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:24:30.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:24:52.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:24:52.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:25:14.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:25:14.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:25:26.820 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:25:26.817+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c0224f0 msgr2=0x7fd98c031d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:25:36.201 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:25:36.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:25:58.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:25:58.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:26:17.769 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:54:04.701 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.785 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:54:04.718 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.838 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:54:04.742 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.842 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:54:04.747 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.843 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:54:04.776 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.850 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:54:04.782 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.855 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:54:04.788 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.929 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:54:04.861 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:17.957 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:54:04.861 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:26:20.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:26:20.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:26:42.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:26:42.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:27:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:27:04.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:27:26.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:27:26.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:27:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:27:46.821+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c040490 msgr2=0x7fd98c0408a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:27:46.829 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:27:46.828+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] conn(0x7fd98c03bee0 msgr2=0x7fd98c0362f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:27:48.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:27:48.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:28:10.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:28:10.200+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:28:32.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:28:32.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:28:54.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:28:54.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:28:56.844 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:28:56.842+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c00fa20 msgr2=0x7fd98c00fe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:29:16.202 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:29:16.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:29:21.849 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:29:21.848+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c0224f0 msgr2=0x7fd98c031d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:29:24.524 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:57:11.456 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.531 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:57:11.463 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.541 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:57:11.473 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.544 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:57:11.477 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.556 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:57:11.460 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.585 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:57:11.489 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.666 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:10:57:11.599 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.722 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:10:57:11.655 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:24.845 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:10:57:11.749 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:29:38.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:29:38.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:30:00.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:30:00.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:30:22.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:30:22.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:30:44.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:30:44.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:31:06.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:31:06.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:31:28.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:31:28.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:31:50.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:31:50.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:32:12.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:32:12.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:32:34.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:32:34.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:32:56.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:32:56.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:33:18.203 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:33:18.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:33:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:33:40.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:34:02.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:34:02.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:34:24.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:34:24.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:34:35.258 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:02:22.190 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.269 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:02:22.202 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.277 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:02:22.209 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.282 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:02:22.186 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.287 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:02:22.219 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.295 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:02:22.199 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.465 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:02:22.396 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.493 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:02:22.424 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:35.578 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:02:22.478 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.538 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:02:26.470 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.540 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:02:26.473 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.563 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:02:26.493 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.566 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:02:26.498 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.596 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:02:26.525 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.597 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:02:26.501 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.601 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:02:26.505 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.608 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:02:26.541 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:39.629 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:02:26.533 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:34:46.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:34:46.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:35:08.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:35:08.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:35:30.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:35:30.201+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:35:31.921 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:35:31.916+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c03bee0 msgr2=0x7fd98c0362f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:35:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:35:52.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:36:14.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:36:14.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:36:36.204 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:36:36.203+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:36:58.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:36:58.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:37:20.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:37:20.203+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:37:42.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:37:42.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:38:04.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:38:04.203+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:38:26.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:38:26.202+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:38:45.683 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:06:32.615 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:45.728 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:06:32.660 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:45.748 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:06:32.680 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:45.765 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:06:32.697 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:45.774 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:06:32.678 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:45.803 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:06:32.707 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:45.925 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:06:32.855 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:45.945 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:06:32.877 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:46.003 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:06:32.906 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:38:48.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:38:48.203+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:39:10.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:39:10.203+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:39:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:39:32.203+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:39:47.044 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:07:33.977 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.051 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:07:33.984 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.066 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:07:33.998 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.071 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:07:33.975 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.072 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:07:34.004 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.081 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:07:33.985 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.116 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:07:34.049 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.135 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:07:34.066 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:47.170 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:07:34.074 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:39:54.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:39:54.203+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:40:16.205 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:40:16.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:40:38.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:40:38.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:41:00.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:41:00.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:41:22.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:41:22.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:41:44.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:41:44.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:41:51.965 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:41:51.963+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.79:6810/10214,v1:172.21.15.79:6811/10214] conn(0x7fd98c00f9c0 msgr2=0x7fd98c00fdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:42:06.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:42:06.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:42:28.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:42:28.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:42:50.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:42:50.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:43:01.722 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:10:48.654 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.769 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:10:48.701 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.770 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:10:48.703 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.789 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:10:48.721 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.815 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:10:48.719 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.817 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:10:48.721 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.877 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:10:48.804 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.938 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:10:48.869 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:01.953 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:10:48.856 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:43:12.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:43:12.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:43:34.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:43:34.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:43:56.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:43:56.204+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:44:18.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:44:18.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:44:40.206 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:44:40.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:44:58.682 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:12:45.614 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.691 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:12:45.624 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.702 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:12:45.634 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.708 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:12:45.640 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.713 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:12:45.617 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.731 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:12:45.634 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.785 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:12:45.717 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.822 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:12:45.754 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:44:58.946 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:12:45.850 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:45:02.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:45:02.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:45:24.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:45:24.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:45:46.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:45:46.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:46:08.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:46:08.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:46:30.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:46:30.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:46:52.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:46:52.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:47:14.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:47:14.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:47:27.010 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:47:27.008+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c03bee0 msgr2=0x7fd98c0362f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T03:47:36.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:47:36.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:47:58.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:47:58.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:48:20.207 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:48:20.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:48:42.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:48:42.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:49:04.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:49:04.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:49:26.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:49:26.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:49:48.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:49:48.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:50:10.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:50:10.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:50:12.932 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:17:59.864 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:12.939 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:17:59.872 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:12.952 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:17:59.884 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:12.956 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:17:59.888 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:12.957 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:17:59.860 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:12.965 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:17:59.869 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:13.161 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:18:00.093 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:13.258 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:18:00.162 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:13.268 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:18:00.200 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:50:32.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:50:32.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:50:54.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:50:54.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:51:16.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:51:16.205+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:51:23.778 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:19:10.710 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.807 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:19:10.739 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.829 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:19:10.761 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.848 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:19:10.778 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.883 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:19:10.787 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.885 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:19:10.789 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.912 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:19:10.844 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.942 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:19:10.874 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:23.951 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:19:10.855 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:51:38.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:51:38.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:52:00.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:52:00.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:52:22.208 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:52:22.207+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:52:44.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:52:44.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:53:06.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:53:06.207+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:53:28.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:53:28.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:53:50.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:53:50.207+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:54:12.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:54:12.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:54:34.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:54:34.207+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:54:56.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:54:56.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:55:18.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:55:18.207+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:55:24.427 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:23:11.359 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.431 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:23:11.363 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.458 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:23:11.362 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.468 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:23:11.397 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.469 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:23:11.369 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.472 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:23:11.404 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.608 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:23:11.539 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.644 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:23:11.574 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:24.676 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:23:11.578 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:55:40.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:55:40.206+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:56:02.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:56:02.207+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:56:24.209 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:56:24.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:56:46.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:56:46.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:57:07.112 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:57:07.110+0000 7fd9ab6fb700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6800/10148,v1:172.21.15.199:6801/10148] conn(0x7fd98c024b40 msgr2=0x7fd98c03b460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T03:57:08.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:57:08.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:57:30.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:57:30.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:57:37.154 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:57:37.151+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6806/10150,v1:172.21.15.199:6809/10150] conn(0x7fd98c040de0 msgr2=0x7fd98c0411f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T03:57:52.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:57:52.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:58:14.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:58:14.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:58:36.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:58:36.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:58:58.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:58:58.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:59:20.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:59:20.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:59:42.210 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T03:59:42.208+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T03:59:45.922 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:27:32.854 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:45.929 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:27:32.861 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:45.961 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:27:32.892 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:45.976 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:27:32.879 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:45.983 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:27:32.886 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:45.984 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:27:32.916 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:46.065 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:27:32.997 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:46.117 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:27:33.020 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T03:59:46.128 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:27:33.060 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:04.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:00:04.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:00:26.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:00:26.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:00:34.470 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:28:21.402 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.475 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:28:21.407 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.483 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:28:21.415 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.486 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:28:21.418 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.490 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:28:21.393 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.497 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:28:21.401 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.527 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:28:21.459 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.537 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:28:21.469 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:34.587 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:28:21.491 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:00:48.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:00:48.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:01:10.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:01:10.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:01:32.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:01:32.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:01:54.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:01:54.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:02:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:02:16.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:02:24.326 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:02:24.324+0000 7fd9aa6f9700 0 -- 172.21.15.79:0/250202870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8e4009c30 msgr2=0x7fd8e400c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-12T04:02:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:02:38.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:03:00.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:03:00.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:03:22.211 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:03:22.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:03:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:03:44.209+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:04:06.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:04:06.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:04:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:04:28.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:04:50.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:04:50.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:05:12.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:05:12.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:05:34.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:05:34.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:05:47.202 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:33:34.134 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.211 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:33:34.143 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.219 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:33:34.151 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.224 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:33:34.156 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.228 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:33:34.132 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.235 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:33:34.137 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.422 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:33:34.354 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.445 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:33:34.377 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:47.514 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:33:34.418 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:05:56.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:05:56.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:06:18.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:06:18.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:06:40.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:06:40.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:07:02.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:07:02.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:07:24.212 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:07:24.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:07:46.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:07:46.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:08:02.252 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:08:02.249+0000 7fd9aaefa700 0 -- 172.21.15.79:0/259216681 >> [v2:172.21.15.199:6816/10151,v1:172.21.15.199:6817/10151] conn(0x7fd98c03bee0 msgr2=0x7fd98c0362f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-12T04:08:07.732 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:35:54.664 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.737 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:35:54.670 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.748 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:35:54.680 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.789 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:35:54.721 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.825 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:35:54.757 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.835 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:35:54.738 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.838 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:35:54.742 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.866 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:35:54.798 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:07.944 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:35:54.848 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:08:08.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:08:08.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:08:30.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:08:30.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:08:52.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:08:52.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:09:14.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:09:14.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:09:36.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:09:36.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:09:58.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:09:58.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:10:20.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:10:20.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:10:42.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:10:42.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:10:59.827 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:38:46.759 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:10:59.831 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:38:46.763 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:10:59.851 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:38:46.754 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:10:59.878 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:38:46.782 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:10:59.929 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:38:46.860 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:10:59.937 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:38:46.868 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:10:59.963 INFO:tasks.ceph.mon.a.smithi079.stderr:==00:11:38:46.895 9281== Warning: unimplemented fcntl command: 1036 2019-12-12T04:11:00.131 INFO:tasks.ceph.mon.c.smithi079.stderr:==00:11:38:47.062 9282== Warning: unimplemented fcntl command: 1036 2019-12-12T04:11:00.161 INFO:tasks.ceph.mon.b.smithi199.stderr:==00:11:38:47.064 8846== Warning: unimplemented fcntl command: 1036 2019-12-12T04:11:04.213 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:11:04.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:11:26.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:11:26.210+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:11:48.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:11:48.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:12:10.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:12:10.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:12:32.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:12:32.212+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:12:54.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:12:54.211+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:13:16.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:13:16.212+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:13:38.214 INFO:teuthology.orchestra.run.smithi079.stderr:2019-12-12T04:13:38.212+0000 7fd97b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-12T04:13:49.754 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-12-12T04:13:49.754 DEBUG:teuthology.task.console_log:Killing console logger for smithi199 2019-12-12T04:13:49.755 DEBUG:teuthology.task.console_log:Killing console logger for smithi079 2019-12-12T04:13:49.755 DEBUG:teuthology.task.console_log:Killing console logger for smithi199 2019-12-12T04:13:49.756 DEBUG:teuthology.task.console_log:Killing console logger for smithi079 2019-12-12T04:13:49.756 DEBUG:teuthology.exit:Finished running handlers